This wiki is obsolete, see the NorduGrid web pages for up to date information.


From NorduGrid
Jump to navigationJump to search


NOTE: This page is out of date and is kept for purely historical reasons.
Janitor is no longer part of ARC.

This page shall help becoming familiar with the Janitor, i.e. the series of scripts, files and (already applied) patches to A-REX to allow the dynamic addition of runtime environments to the system. As a very quick summary, there are

  • Catalogs describing resources available for installation
    • BaseSystems - what the system is using if run with no RTEs
    • MetaPackages - what shall somehow be achieved to be installed
    • Packages - the explicit package (depending on some base system) achieving it
  • One should know
    • Catalogs are completely trusted
    • Only tar-based packages are available for the time speaking
    • Both automated invokation via the A-REX are possible and manual invocation through the command line interface

Further documentation:

  • Janitor handbook
  • janitor(8) man page


A typical Debian installation comes with a series of packages (try "dpkg -l nordugrid-arc*" or rpm -q -a | grep ^nordugrid-arc) of which only

 ii  nordugrid-arc-nox-janitor 1.1.0~rc3.1-1             ARC dynamic installation of runtime environments

signs responsible for the dynamic runtime environments. A closer inspection shows

A script to update catalogs from remote sources:


An example RDF Catalog file


The current RDF Schema file that should be shared by all Catalogs.


Extensions to the arc client configuration, i.e. the configuration of the location of the catalogs and runtime environments


Specification of parametesr for Logging


A representation fo the Catalog of runtme environments in Perl:


A series of Perl modules to represent the core functionality of the Janitor.


The man page


The perl script to be executed for performing action with the janitor. One may argue this would need to be located in /usr/bin, but not from the perspective of the A-REX that should executed it. The final word is not spoke here, but today it seems likely that as soon as the A-REX works no longer as root but under a different user interface, a wrapper script will be come back to action with the suid bit set to perform the demands that the non-privileged A-REX will then not be capable doing.


Becoming familiar with the CLI

Once one has understood the CLI, the understanding of the complete system needs to imagine the A-REX using that CLI.


The Janitor expects some basic info about what catalogs to read. Pass it the configuration in /usr/share/arc/janitor/conf/arc.conf as an argument or add its contents to /etc/arc.conf. If you are not using /etc/arc.conf but some XML based configuration, then for now please just create the file with this content or copy the file to that location:

# this should already be specified by current ARC installation

and further down then please add a section like



The first block specifies some basic settings on how the Janitor should work. The second specifies on catalog that should be used to identify installable runtime environments. In a running system, one is likely to know about multiple such catalogs, e.g. some from colleagues, others from some trusted URL on the net.

One is tempted to just copy the complete catalog as file /var/spool/nordugrid/janitor/catalog/knowarc.rdf. As a start, please just do so. But please edit this information as soon as possible to truly reflect what you want to allow to be installed - and make sure the files to be installed are truly available etc ...

Display of Catalog

Of a complete overview on what is going on, perform

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list

you will get


Locally installed runtime environments:

Dynamically installed runtime environments:

Installable runtime environments:

  name:        APPS/HEP/ROOT-
  description: ROOT                   
  supported:   no                     

  name:        APPS/CHEM/GAMESS-20041122-0
  description: Quantum chemistry program for the calculation of SCF wavefunctions ranging from RHF, ROHF, UHF, GVB, and MCSCF.                                                                                                                         
  supported:   no                                                                                                          

  name:        APPS/HEP/ATLAS-
  description: ATLAS                   
  supported:   no                      

  name:        APPS/HEP/CERNLIB-2005
  description: CERNLIB              
  supported:   no                   

  name:        ENV/LANGUAGE/GERMAN
  description: German language environment
  supported:   no                         

  name:        APPS/CAS/SIMICS-2.2.7-1
  description: Computer Architecture Simulator
  supported:   no                             

  name:        ENV/DEVELOPMENT/GCC-4.1
  description: The GNU C compiler, version 4.1
  supported:   no                             

  name:        APPS/BIO/WEKA-3.4.10
  description: WEKA Machine Learning Software
  supported:   yes                           

  name:        ENV/FORTRAN/G95
  description: The GNU Fortran 95 compiler
  supported:   no                         

  name:        APPS/BIO/TFBS-0.5.0
  description: TFBS               
  supported:   no                 

  name:        ENV/DEVELOPMENT/G++-4.1
  description: The GNU C++ compiler, version 4.1
  supported:   no                               

  name:        APPS/BIO/LAGAN-1.2
  description: LAGAN             
  supported:   yes               

  name:        APPS/GRAPH/POVRAY-3.6
  description: The Persistence of Vision Raytracer
  supported:   no                                 

  name:        ENV/PARALLEL-1.0
  description: PARALLEL environment
  supported:   no                  

  name:        ENV/JAVA/JRE-1.5.0
  description: Sun Java Runtime Environment of version 1.5
  supported:   no                                         

  name:        ENV/JAVA/SDK-1.4.2
  description: Java2 Software Development Kit
  supported:   no                            

  name:        ENV/MPI
  description: MPI environments branch
  supported:   no                     

  name:        ENV/LOCALDISK-0000
  description: Advertises the availability of local scratch disk on compute nodes.  The number trailing RE name (here 0000) is the available disk space in MegaBytes.                                                                                  
  supported:   no                                                                                                          

  name:        APPS/MATH/ELMER-5.0.2
  description: Finite Element Software for Multiphysical Problems
  supported:   no                                                

  name:        APPS/STATISTICS/R-2.3.1
  description: R                      
  supported:   no                     

  name:        ENV/JAVA/JRE-1.4.2
  description: Java2 Runtime Environment
  supported:   no

  name:        APPS/BIO/JASPAR-CORE-1.0
  description: JASPAR-CORE
  supported:   yes

  name:        APPS/BIO/WEKA-3.4.8A
  description: WEKA Machine Learning Software
  supported:   yes

  name:        APPS/BIO/AUTODOCK
  description: AutoDock is a suite of automated docking tools.                 It is designed to predict how small molecules, such as substrates           or drug candidates, bind to a receptor of known 3D structure.
  supported:   yes

0 Successfully retrieved information.

This is already impressive. But most of the above are just the regular runtime envivonments that one knows from the NorduGrid RTE page. Those are available as MetaPackages, but with no package implementing them. In currently empty sections above, the MetaPackages would be shown if they were already installed by some means. And there are no jobs currently being registered with any RTE. To only see the installable list of MetaPackages (supported or not), do

 /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list installable

See the janitor(8) man page for further options to display the status.

Performing the automated installation

The automated installation makes sense only when there is a job demanding the availability of the package. Those jobs then have a jobid to identify it and the Janitor uses that for the associated of runtime environments with a job. So, after the job has run, one can then delete the runtime environments that are of no further use. But one could also think of abstract "resons" or almost tangible "projects" that could be given a number, too. So, you would delete a RTE only after that project has ended or there no reason any more to keep it. The registration with a jobid or projectid works like follows:

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf register 0 APPS/BIO/AUTODOCK
1 Sucessfully initialized job.

Now, the Janitor-internal information system lists its system as INSTALLABLE

/usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list rtes | head -10

Locally installed runtime environments:

Dynamically installed runtime environments:

  name:        APPS/BIO/AUTODOCK
  state:       INSTALLABLE
  lastused:    1267096878

There are no locally installed runtime environments, which would be the the classical runtime environments, but we now have a dynamically installed on registered locally that is not installed but 'installable', so the Janitor thinks because it has found a package. Now for the deployment:

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf deploy 0
1 Cannot provide requested RTEs: installation FAILED.

The state of the previously installed runtime environment has changed accordingly.

Dynamically installed runtime environments:
  name:        APPS/BIO/AUTODOCK
  state:       FAILED           
  lastused:    1267096878  

The "lastused" date is the UNIX epoch, i.e. seconds since 1970. We still need to find out why this has not worked. Then the installation fails, the Janitor assumes that the job will be rejected by the grid manager and consequently does not register the job. The information about he failure remains persistent, so subsequent attempts will falter immediately and are announced as such by the information system.

Since there is no job assigned, the failed RTE also cannot be removed, i.e. it is the job that is removed (deregistered) and the RTE then goes (after some expiration time) with it. To remove the failed RTE, we first set its state to removal pending.

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf setstate "REMOVAL_PENDING"  APPS/BIO/AUTODOCK
0 Changed state successfully.

It is still there

Dynamically installed runtime environments:

  name:        APPS/BIO/AUTODOCK
  state:       REMOVAL_PENDING
  lastused:    1267096878

Until we now perform a sweep

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep
[Janitor::Janitor] 2010/02/25 13:17:46 INFO> Removing APPS/BIO/AUTODOCK
0 Sweeping is done.

It is gone now. The challenge now is to either indicate a proper URL to the catalog entry. If it was not your very own catalog entry but some public one then it should have working packages announced not as a local URL but such accessible via http.

Preparation of a dynamic Runtime Environment package

The dynamic runtime environments may come in different flavours. Currently supported are tarfiles with two subdirectories:

  • data - all the bits one expects from the user's perspective
  • config - scripts to set the PATH and other environment flags
    • install - perform post-processing after the data directory is copied to its destination
    • remove - executed just prior to the removal of the files from the data directory.
    • runtime - executed for every single job, repeatedly for multiple stages of job execution

The install and remove scripts, the latter commonly empty, are unique to the dynamic runtime environments. The runtime scripts are also required for the classical ARC runtime environments.

Building a very basic dRTE

As a playground binary let us prepare a HelloWorld application that prints exactly that.

[ -d data ] || mkdir data
cat <<EOHELLOWORLD > data/HelloWorld
echo 'Hello World!'
chmod +x data/HelloWorld

and there is not much to do for the install files etc..

[ -d control ] || mkdir control
echo '#!/bin/sh' > control/install 
echo 'echo install invoked as: $0 $*' >> control/install
echo '#!/bin/sh' > control/remove
echo 'echo "Nothing extra to be done for removal."' >> control/remove
echo '#!/bin/sh' > control/runtime
echo 'export PATH=$PATH:%BASEDIR%' >> control/runtime
echo 'echo runtime invoked as: $0 $*' >> control/runtime

The BASEDIR string will be substituted with the final installation directory.

Then this needs to be packed:

GZIP=-9 tar czvf HelloWorldRTE.tar.gz control data

a be made known to the Catalog.

Extending the Catalog

The Catalog is an RDF file. This way we can talk about some object in a distributed manner, i.e. we are not depending on the tree like structure. The objects/subjects you phrase triplets over (with predicates) all have their unique identifies, called URIs. Those look like URLs but do not need to exist.

To extend the Catalog, we introduce the concept of a HelloWorld package

cat <<ADDTHISTOTHECATALOG >> /var/spool/nordugrid/janitor/catalog/knowarc.rdf
<kb:MetaPackage rdf:about="&kb;MetaPackage/demo/hello/world"
       <kb:description>A minimalistic dynamic Runtime Environment, featuring the exutable script HelloWorld.</kb:description>
       <kb:instance rdf:resource="&kb;package/tar/helloworld-0.0"/>

and then inform about he availability of a new TarPackage.

cat <<ADDTHISTOTHECATALOG >> /var/spool/nordugrid/janitor/catalog/knowarc.rdf
<kb:TarPackage rdf:about="&kb;package/tar/helloworld-0.0"
       <rdfs:label>HelloWorld </rdfs:label>
       <kb:basesystem rdf:resource="&kb;BaseSystem/debian/etch"/>

It is a matter of the individual sites to accept or deny claims for a particular base system.

The package will then be listed as

  name:        APPS/DEMO/HELLO/WORLD
  description: A minimalistic dynamic Runtime Environment, featuring the exutable script HelloWorld.
  supported:   yes

Registration and Deployment

This time, registration was already

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf register 1 APPS/DEMO/HELLO/WORLD
janitor: Warning: running with group root.                                                                            
[Janitor::Janitor] 2010/02/28 02:01:13 INFO> 1: registering new job (APPS/DEMO/HELLO/WORLD)                           
[Janitor::Janitor] 2010/02/28 02:01:13 INFO> 1: sucessfully registered job                                            
1 Sucessfully initialized job.

So, registration was successful:

Toshiba:/home/moeller/t# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list

  jobid:       1
  created:     1267318873
  age:         16 seconds
  state:       PREPARED  
  rte:         APPS/DEMO/HELLO/WORLD

And so is the deployment:

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf deploy 1
[Janitor::Janitor] 2010/02/28 02:02:07 INFO> 1: use basesystem "debian::etch_tar" for dynamically installed rte
[Janitor::Janitor] 2010/02/28 02:02:07 INFO> installing package (Catalog: [NONAME])
[Janitor::Janitor] 2010/02/28 02:02:08 INFO> 1: sucessfully initialized job
0 Sucessfully initialized job.

Some closer inspection:

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list
janitor: Warning: running as user root.                                                   
janitor: Warning: running with group root.                                                

  jobid:       1
  created:     1267318873
  age:         6 minutes 
  state:       INITIALIZED
  rte:         APPS/DEMO/HELLO/WORLD

Locally installed runtime environments:

  name:        APPS/DEMO/HELLO/WORLD

Dynamically installed runtime environments:

  name:        APPS/DEMO/HELLO/WORLD
  state:       INSTALLED_A          
  lastused:    1267318928

The UNIX epoch to display the last usage seems improvable. And, we should learn about where it is installed, well, that we know since we know where grid runtime environments are to be installed:

# grep runtimedir /usr/share/arc/janitor/conf/arc.conf

In that directory, the name of the app indicates the location of the configuration script, just like for the regular runtime environments

# find /var/tmp/arc/runtime/config

But that does not necessarily indicate on the whereabouts of the installed runtime directory:

# grep installationdir /usr/share/arc/janitor/conf/arc.conf

This way, the classical and dynamical runtime environments are clearly separated:

# find /var/spool/nordugrid/janitor/runtime

To remove the dRE again, one dergisters the job

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf  remove 1
[Janitor::Janitor] 2010/02/28 02:21:01 INFO> removing job 1
[Janitor::Janitor] 2010/02/28 02:21:01 INFO> 1: removed
0 Removed.

"0" is the exit status, here indicating no error.

The job is gone, but the RTE still present:

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list 


Locally installed runtime environments:

   name:        APPS/DEMO/HELLO/WORLD

Dynamically installed runtime environments:

  name:        APPS/DEMO/HELLO/WORLD
  state:       INSTALLED_A          
  lastused:    1267320061           

The job is removed with sweep,

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep
[Janitor::Janitor] 2010/02/28 03:00:56 INFO> Removing APPS/DEMO/HELLO/WORLD
[Janitor::Janitor] 2010/02/28 03:00:56 INFO> Removing HelloWorldRTE.tar.gz
0 Sweeping is done.

with sweep --force if one does not want to respect minimal lifetimes or other issues.

# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep --force
0 Sweeping is done.

Working with Packages from Linux distributions

Retrieval of packages

One can get Packages from Linux distributions directly from the ftp or http servers (mirrors) of the distribution. Alternatively, particularly if interested in experimenting also locally with the software, one may prefer the local installations, first:

# apt-get install clustalw autodock autogrid boxshade

and then save the .debs in a local folder

# mkdir debs
# cp /var/cache/apt/archives/*.deb debs/

before removing them again

# apt-get clean

This is somehow similar for all the distributions.


One could think of various ways to integrate Debian or another distribution with grid computing:

  • chroot environments (like Debian build daemons)
  • virtual machines to be configured at runtime
  • ... ?

We have played with the above, but beyond reporting that it works, we have not investigated that further. For this introductory round, we present the conversion of Debian packages into ARC-typical runtime environments.

editing not yet complete -- Steffen


This shall obviously happen prior to the removal of the runtime enironment. The exact name of the installation directory is not known in advance and not guaranteed to remain the same between job invocations. The executing scripts completely depends on environment variables to be set by the runtime script.

$ cat /var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/runtime
export PATH=$PATH:/var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg
echo runtime invoked as: $0 $*

The pkg directory is our former data directory:

$ ls /var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg

and when exuting the runtime script (performed automatically by the grid backend scripts), it indeed works:

$ export PATH=$PATH:/var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg
$ HelloWorld
Hello World!

Integration with A-REX and ISIS

Verification of this paragraph is still pending: If the configuration files of the Janitor are performed correctly, then (upon a manual invocation of the Janitor) there will be no difference to the classical runtime environments. When ARC is compiled with the option to support the Janitor, then this will be performed with no further configuration.

The available packages are also forwarded to the grid information system. This goes beyond what was available for classical runtime environments, i.e. packages that are available but not yet installed.