ETICS overview, December 2008

What ETICS is and what it is not

ETICS is a framework for building and testing software. It is a collection of tools and services (infrastructure) aimed at automatization of building, testing and report generation processes. It is not replacement for Bug tracking system nor the Version Control System though one of provided services is repository for built software packages and corresponding test results.

ETICS does not provide any testing software itself (be it cppunit or tools for static code analysis and obviously not for functional tests), It only provides hooks for such testing tools which are then to be trigerred in customary configuration (via testing scripts) during build and testing itself. ETICS is able to run any user defined test (be it a script or binary) and judges its success based on the returned value. Reports are generated in the form of static XML and HTML pages. Supported and custom defined metrics calculations can be included in the reports as well.

Quality model

With respect to testing the system rely on A-QCM which stands for Automated Quality Certification Model, which is customized quality standard based on several ISO procedural standards and ISO 9126 for definition of software quality. Given the fact that A-QCM has been developed considering several ISO standards thus slots for description, structure and names of all relevant quality aspects (metrics) should be trackable. Thus most of existing metrics should be possible to tackle (this inevitably depends only on the quality and coverage of test cases). Put another way quality metrics that ETICS is able to calculate are defined within A-QCM, plus user has a possibility to define unlimited number of custom metrics.

Implementation

ETICS client tools are written in python, thus it can run on any platform where python is supported (>v2.2). With respect to testing tools (cppunit, tools for static code analysis, coding rules compliance), I've already mentioned that these are not part of ETICS neither, thus they can be used on platforms where they are supported.

Crucial part of ETICS consists of its Web portal (https://etics.cern.ch/eticsPortal/) which serves for builds and test configurations. The build system currently supports 57 platforms (specific combinations of Distribution, architecture and compiler). From the platforms for which ARC nightly builds are performed, recent and older Fedoras (1,2,4,7,8,9), redhat9 and ubuntu7 (x86_64) are missing. But there some others platforms instead: Yellowdog (ppc 32/64 bit), Debian3, Solaris10 (x86), Darwin (x86/ppc), Win32_vc71, aix52 (ppc).

Comparison ETICS vs Anders' build system + ARC testing framework

I like Anders' build system more ;-). The reason is that I found it more open (ETICS does not advertise its solution for services for download anywhere) in the sense that everybody can download it and straightforwadly use it on its own ARC cluster for building any type of software, while this is possible also with ETICS, it is just will of somebody else how much I will be given (that resembles me a difference between giving a fish and teaching how to fish).

With respect to testing framework, the ETICS solution is considerably more elaborated, especially with its ability of producing complex reports and metrics calculations upon test results. If I found a way how to use these tools (especially testManager) locally I would definitely consider its use in our testing framework.

Based on what I've found till now the answer to the question: "How do you define tests?" is that the tests can be defined within custom scripts. The success or failure of a particular test is judged based on return value (0 for success, other values for failures). The information on user tests, its order of execution and dependencies among them can be submitted via web portal. The information are stored which generates some XML descriptions of user defined tests (and dependencies among them). This description is then stored on the ETICS system accessible via portal and any user with access can than run specified tests either on local computer or remotely on the grid build system (the ETICS web portal stores only information on what was/should be performed and all obtained results, built packages and generated reports).

Drawbacks

I did not find any documentation on the ways to configure building and testing locally without need of making use of ETICS client tools without need for registration with ETICS. Thus even if it is possible to run builds and tests locally, the registration and actual use of ETICS services seem to be inevitable. (After I've figured out that the registration is probably inevitable I've asked for registration https://etics.cern.ch:8443/eticsAdmin/public/registration/requestRegistration.jsp, this registration should allow me to create test projects within the system ).

Discussed limited number of platform support is not an issue with ETICS it is specific to gLite. Based on what I've learned so far also discussed problem with global rebuilds is not a problem in ETICS it is be possible to configure it in a way preventing such a need.

Sources of information

1. The official website of the project http://etics.web.cern.ch/etics , contents of which dramatically improved over last three weeks from almost none to quite coherent one now.
2. ETICS phase one website http://project-eu-etics1.web.cern.ch/project-eu-etics1/index.html (project ended in december 2007).
3. Project wiki: https://twiki.cern.ch/twiki/bin/view/ETICS/WebHome
4. Manual: https://edms.cern.ch/file/795312/2.0.1/ETICS-User_Manual-latest.pdf