ETICS

The purpose of this page is to introduce an interested person into ETICS. With special emphasis on its role in EMI with practical examples using ARC configurations for EMI releases.

Introduction

ETICS is a framework for building and testing software. It is a collection of tools and services (infrastructure) aimed at automation of building, testing and report generation processes.

The build process within ETICS works like this:

1. Start the build machine with the required operating system.
2. Install all the dependencies
3. Compile the software and package it
4. Test the software
5. Publish results

ETICS configurations specify and control most of that process. The actual run of ETICS build/test can be further modified during the submission of a given configuration by specifying a concrete platform, requesting creation of repository or (un)checking particular options relevant to creation of packages (e.g. mock repackaging).

Get to know it

There is a very nice set of video tutorials available, providing a really gentle introduction to the ETICS, I would start with them

http://etics.web.cern.ch/etics/training/tutorials/index.htm

( available also on youtube )

And here is link to more EMI centric training lead by EMI - ETICS chief developer (Lorenzo Dini)

http://connect.ct.infn.it/p10496307/?launcher=false&fcsContent=true&pbMode=normal

All the relevant general documentation on ETICS is linked from here

http://etics.web.cern.ch/etics/

Get access to it

Register at https://etics.cern.ch/eticsPortal/ . The registration is quite straightforward (you should have your certificate installed in the browser). The process of is also described in the first video of the tutorial series. The access is usually granted within 24 hours.

To start practical work on relevant subsystems and configurations one needs to get proper permissions. In the case of ARC, the relevant subsystems are "arc" and "emi/arc". "arc" served so far as a playground for trying out the system while "emi/arc" contains also production configurations used for EMI releases. The permissions are to be granted and managed by the project (or subsystem) administrator.

Building ARC packages for EMI

The only way to have packages in EMI releases is to build it successfully in ETICS.

Configurations' maintainer workflow

The standard workflow of the ARC ETICS person in EMI:

1. Attending weekly EMT meetings ([ https://twiki.cern.ch/twiki/bin/view/EMI/EMT Engineering Management Team ])
2. Keeping an eye on ARC SVN for new tags and communicating with Nordugrid release coordinator (Daniel Johansson) at least keeping an eye his announcements at the ng-disc
3. Creating/updating ARC configurations in the ETICS (emi/arc) subsystem
   1. submitting builds to check the things are building properly
   2. occasionally running tests himself (at least installation) but more often pinging testers or announcing availability of packages to test on mailing list (arc-emi-all)
   3. communicating with EMI release manager if/when the configurations are ready for him to use in overall project configuration
4. Informing about relevant changes or problems arc-emi-all list or Certification manager for ARC product teams.

Mock in ETICS

The proper way to prepare binaries is to build them in pristine environment. Nordugrid has traditionally used Mock for the purpose (standard way adapted also by Fedora, EPEL and other RH based distros). That solution has been adapted by EMI and implemented into its building tool ETICS (which caused some headaches and delays with EMI-0 and EMI-1, but finally the feature is there ;-).

To build binaries with Mock in ETICS do the following: when submitting build via the web portal click on the "more options" in the "Build Execution" section and put --repackage <config> in the "Custom Build Options" field (in the case of command line client just put --repackage <config> as an option to the etics-build command).

In the case of EMI1 updates the relevant <config> is either "emi-1-sl5-x86_64" or "emi-1-sl5-i386" (new configs are prepared for new release candidates and major releases).

Check EMI packaging policy for details

Practical Examples

To build successfully configurations under "emi/arc" subsystem against EMI dependencies one has to check one option during the build (well, whatever includes building of the code). The label of the option in the web portal is called "Propagate environment and properties from" in command line client it is "--project-config" option. The top-level configs that ARC should build successfuly with are "emi_R_1_rc", "emi_R_1_prod" and "emi_B_1_dev". Check section "ETICS-project configurations" of the following wiki page maintained by the Release Manager https://twiki.cern.ch/twiki/bin/view/EMI/EMI1Updates for details on purpose and differences of those three configurations.

Start to play with production configurations

I would recommend to start by looking at the production ARC configurations that were used for EMI-1 release. Those are locked and can not be edited anymore. But you can clone them (form of copying) and then even modify it and play around.

This is how you can do it when using web interface ( configuration tab at https://etics.cern.ch/eticsPortal/#legBs ). Just go into emi -> arc subsystem. After clicking on the arc, you should see several subsytem configuration in the lover-left corner segment of the browser. The configuration that was used for the EMI-1 release is

emi-arc_R_1_0_0_3

right click on it and select "Add to the workspace". It should appear in the central segment of the interface. Expand it you can check how it is composed. The subconfigurations that are of interest to you are the following four ones

emi-nordugrid-arc-compat_R_1_0_0_3 emi-nordugrid-arc-doc_R_1_0_0_2 emi-nordugrid-arc-meta_R_1_0_0_2 emi-nordugrid-arc_R_1_0_0_2

the other six configurations were added by the release manager for some organizational reasons that are unknown to me (thay do not produce any packages).

You can clone (choose proper new name) and play with any of those configurations. To add the newly cloned configuration in the workspace you can either go and find it in the respective component (e.g. emi-nordugrid-arc_R_1_0_0_2 resides under emi->arc->arc1 ) and add it from there or you can click red loop with green plus sign in the upper right corner.

To make a build with a new tag, there is only one field that should be changed in the configuration: the "tag" field on the configuration toplevel board.

The current configurations are sl5 specific. For sl6 or any other RPM based distro the required changes are trivial. For e.g. debian some more editing will be required.

Remember you can not edit "locked" configurations. Make a copy if you want to play.

Preparing configurations for EMI-1 updates

Let's have the following situation. Since there were found and fixed some bugs in the ARC components in the EMI-1 release, the ARC release manager tagged new release candidate for arc1 components.

Let's suppose the new tag resides under arc1/tags/1.0.1rc4 in the nordugrid svn repository.

Sure we want to have the updated packages also in EMI-1 repositories. There are the following things now that has to be done:

• Create new configurations for arc1 component for which there is a new tag. In our concrete example just clone emi-nordugrid-arc_R_1_0_0_2 into something like emi-nordugrid-arc_R_1_0_1_4 and change the tag field from arc1/tags/1.0.0 into arc1/tags/1.0.1rc4. That's all.
• Create a new subsystem configuration for whole arc subsystem. In our concrete example just clone emi-arc_R_1_0_0_3 into something like emi-arc_R_1_0_1_4. Then edit it so that arc1 component uses the newly created emi-nordugrid-arc_R_1_0_1_4 configuration. That's it.
• Check that the new subsystem configuration builds properly on supported platforms (sl5_ia32_gcc412EPEL and sl5_x86_64_gcc412EPEL for EMI1) against proper project configs (e.g. emi_B_1_dev and emi_R_1_rc).
• If builds are OK, ask the EMI release manager (Cristina Aifitimiei) to update overall EMI configuration
• If builds are OK also in the nightly builds you can lock the new configuration and also the subsystem configuration so that no one can edit nor delete it anymore
• Once the packages are in the testing repo, you announce it to the testing team (via arc-emi-all mailing list) and wait for the results.

That's pretty much it. The one who is in charge of certification process then, depending on the testing results, asks EMI release manager to put the new subsystem configuration into production or you (you ask yourself if you happen to be certification manager as well ;-) to repeate the process with the new release candidate or the final tag.

More information on the the whole certification process within EMI can be found in this checklist.

Presentations, Reports and Memos

Here can be found various docs on the topic of ETICS and integration of ARC into it.

1. ARC packages in EMI, presented on NorduGrid 2011, Oslo, May 2011
2. ARC Experiences with ETICS, April 2009
3. ARC in ETICS, introduction, NorduGrid technical meeting, Uppsala, April 2009
4. Question and Answers from f2f meeting with ETICS team, February 2009
5. ETICS overview, December 2008