Question and Answers from f2f meeting with ETICS team, February 2009

These are collected answers from f2f meeting with ETICS team in Budapest, February 2009.

Fair Usage Policy

question

Given the very nature of software testing, it is not difficult to arrange tests that would swallow any resource given. Is there any fair usage policy (and its enforcement) defined within ETICS? How would one prevent possible unavailability of the services due to over usage by some of the users?

ETICS has not mechanism to control/quote resource consumption of resources by user. The task in general is left to the resource owner/provider. On the other hand any Project in ETICS is able to add his own dedicated resources (individual machines) to the pool of build/test machines already available. Those machines will be then exclusively available by the project itself.

Terminology clarification

question

There are currently ~64 platforms registered within ETICS (returned from etics-list-platform binary), but directly usable is only subset of those platforms. What does the term "registered" mean? What does it take to make a platform registered and registered platform usable?

answer

To register a platform means really just to add an item in the ETICS database. Once the platform is registered dependencies and configurations for it can be edited and stored. Once the platform is available physically in the pool of machines remote builds and tests can be performed or users can use their own machines of corresponding platform for this making use of configurations and dependencies stored in ETICS database.

Platform availability

question

Is it true that some platforms can be more available than others? Are the testing machines virtual?

answer

The machines are both physical and virtual (larger part). All machines are running constantly (no dynamical loading) so some platforms are present in higher number then others (e.g. ~20 SL4 compared to 1 Mac OS X) but in the case of need required platforms can be added. User can even submit images of virtual machines that he would like to have in a pool.

External dependencies, procedure

question

What does it take to add a new external dependency within ETICS? Are external dependencies added automatically upon request?

answer

The process requires manual intervention from ETICS people. Adding an external dependency within means also that its maintanance will be performed by ETICS itself (so that security updates etc. for the package are integrated by themselves ever since). If user does not want to wait for integration of required external dependency from ETICS, he can add it within his own project in that case he is responsible for its maintainance.

External dependencies, formats

question

Can be external dependencies defined only as a binary packages?

answer

No for the package field for the requested component does not have to contain a binary necesserily. If it's needed ETICS will rebuild the code for requested platforms.

External dependencies, ETICS internals

question

Can be as dependencies defined also software from others projects (others then 'externals')?

answer

Yes, dependencies can be specified among any components within ETICS tree.

Partial rebuilds

question

Is there any simple way that would allow to perform only partial rebuilds of the source tree (to speed up a process of recompilation after changing just a few lines in one of source files)?

answer

There is always way on how to do it but how laborious it is depends on the structure of source tree with which particular Project works (its granularity).

Quality metrics specifications

question

There are several quality metrics predefined within ETICS. How the user specifies to which of those relates particular test?

answer

Identification of tests and also corresponding metrics trigerred is indentified by a plugin that invokes particular test. Those plugins take care also for creation of xml and html reports upon tests performed. In general ETICS recognize two types of tests: build time tests (unit tests, code coverage, cyclomatic complexity, find flaws ...) and deployment tests (in general functionality and performance tests). The first part of tests uses standard command line tools that are invoked via plugins. Since these tools are standard and widely used the chances are that plugins are already written and can be readily used. The second type of tests is also to be invoked via plugins but this is more tricky since it is up to user to write it (there is published API). The effort needed to prepare custom plugins is recognized by ETICS as the issue and they are working on simplification of the task.

With respect with the deployment tests there is an effort to standardize way on how to perform really complex ones, including several server and client machines. The results are to be expected by summer.

New metrics

question

User is said to have possibility to define his own quality metrics. How is it done?

answer

See answer to previous question: via plugins.

Metrics internals

question

From source code of TestManager one can read about xml file that is used to describe tests (name it), specify order of execution of tests and dependencies between tests. Is that file generated automatically? How does this xml file relate to the testing scripts that can be uploaded and executed in testing steps (submitted through "Test Commands" in configuration editing part of ETICS WebPortal)?

answer

It is true that TestManager was originally standalone program that is currently part of ETICS and in principle could be usable by itself but it is not supported way. User/tester should not need to be exposed to its internals. The mentioned xml file is created automatically by internal processes within ETICS based on particular test plugin design and corresponding test specifications.

building user privileges

question

What is the account and corresponding privileges under which compilation and testing within ETICS runs?

answer

Per default no extra privileges, but build/test run can be specified to be ran under root account (internally this requirement will automatically limit pool of execution machines to the virtual ones).

Functional tests

question

Then we would be very interested in how to run tests involving several machines. How far is it? Is there any predefined procedure for running such tests?

answer

This is ongoing effort, there are already working examples but there is currently a lot of unfinished things. The production state of this feature is to be expected by June of this year.

We have discussed also many other technical details (I won't go into it now) and we exercised with some practical examples on configuration of build execution of ARC within the system.

Conclusion

The message is that, as of now it is possible to use ETICS for building, packaging and static testing. Functionality testing is not yet straightforward, if that feature would be available and configurable in an easy way than it would be something really special (it is not available in any other similar tool).

As I see it, the advantage that of ETICS in comparison with other tools for continuous integration and release management lies in shared environment it provides so that collaborating/competing projects keeps their innards laid bare which can be very healthy.

Some of these other tools might be better in particular tasks (do the right thing right e.g. Cruise or Hudson). Given the fact that UMD is probably going to be cooperation of relatively loosely coupled bodies, use of ETICS could prevent some negative aspects of such looseness. (Everybody is able to see everything, try new things immediately from one place).