This wiki is obsolete, see the NorduGrid web pages for up to date information.
Janitor
|
|
NOTE: This page is out of date and is kept for purely historical reasons. |
This page shall help becoming familiar with the Janitor, i.e. the series of scripts, files and (already applied) patches to A-REX to allow the dynamic addition of runtime environments to the system. As a very quick summary, there are
- Catalogs describing resources available for installation
- BaseSystems - what the system is using if run with no RTEs
- MetaPackages - what shall somehow be achieved to be installed
- Packages - the explicit package (depending on some base system) achieving it
- One should know
- Catalogs are completely trusted
- Only tar-based packages are available for the time speaking
- Both automated invokation via the A-REX are possible and manual invocation through the command line interface
Further documentation:
- Janitor handbook
- janitor(8) man page
Overview
A typical Debian installation comes with a series of packages (try "dpkg -l nordugrid-arc*" or rpm -q -a | grep ^nordugrid-arc) of which only
ii nordugrid-arc-nox-janitor 1.1.0~rc3.1-1 ARC dynamic installation of runtime environments
signs responsible for the dynamic runtime environments. A closer inspection shows
A script to update catalogs from remote sources:
/usr/share/arc/janitor/catalog/old/update_catalog.pl
An example RDF Catalog file
/usr/share/arc/janitor/catalog/knowarc.rdf
The current RDF Schema file that should be shared by all Catalogs.
/usr/share/arc/janitor/catalog/knowarc.rdfs
Extensions to the arc client configuration, i.e. the configuration of the location of the catalogs and runtime environments
/usr/share/arc/janitor/conf/arc.conf
Specification of parametesr for Logging
/usr/share/arc/janitor/conf/log.conf
A representation fo the Catalog of runtme environments in Perl:
/usr/share/arc/perl/Janitor/Catalog/*.pm
A series of Perl modules to represent the core functionality of the Janitor.
/usr/share/arc/perl/Janitor/*.pm
The man page
/usr/share/man/man8/janitor.8.gz
The perl script to be executed for performing action with the janitor. One may argue this would need to be located in /usr/bin, but not from the perspective of the A-REX that should executed it. The final word is not spoke here, but today it seems likely that as soon as the A-REX works no longer as root but under a different user interface, a wrapper script will be come back to action with the suid bit set to perform the demands that the non-privileged A-REX will then not be capable doing.
/usr/lib/arc/janitor
Becoming familiar with the CLI
Once one has understood the CLI, the understanding of the complete system needs to imagine the A-REX using that CLI.
Configuration
The Janitor expects some basic info about what catalogs to read. Pass it the configuration in /usr/share/arc/janitor/conf/arc.conf as an argument or add its contents to /etc/arc.conf. If you are not using /etc/arc.conf but some XML based configuration, then for now please just create the file with this content or copy the file to that location:
# this should already be specified by current ARC installation runtimedir=/var/tmp/arc/runtime/config
and further down then please add a section like
[janitor] enabled="1" logconf="/etc/janitor/log.conf" registrationdir="/var/spool/nordugrid/janitor" installationdir="/var/spool/nordugrid/janitor/runtime" downloaddir="/var/spool/nordugrid/janitor/download" jobexpirytime="7200" rteexpirytime="36" uid="root" gid="0" allow_base="*" allow_rte="*" [janitor/nordugrid] catalog="/var/spool/nordugrid/janitor/catalog/knowarc.rdf"
The first block specifies some basic settings on how the Janitor should work. The second specifies on catalog that should be used to identify installable runtime environments. In a running system, one is likely to know about multiple such catalogs, e.g. some from colleagues, others from some trusted URL on the net.
One is tempted to just copy the complete catalog as file /var/spool/nordugrid/janitor/catalog/knowarc.rdf. As a start, please just do so. But please edit this information as soon as possible to truly reflect what you want to allow to be installed - and make sure the files to be installed are truly available etc ...
Display of Catalog
Of a complete overview on what is going on, perform
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list
you will get
Jobs: Locally installed runtime environments: Dynamically installed runtime environments: Installable runtime environments: name: APPS/HEP/ROOT-4.04.02.6 description: ROOT supported: no name: APPS/CHEM/GAMESS-20041122-0 description: Quantum chemistry program for the calculation of SCF wavefunctions ranging from RHF, ROHF, UHF, GVB, and MCSCF. supported: no name: APPS/HEP/ATLAS-13.0.10.1 description: ATLAS supported: no name: APPS/HEP/CERNLIB-2005 description: CERNLIB supported: no name: ENV/LANGUAGE/GERMAN description: German language environment supported: no name: APPS/CAS/SIMICS-2.2.7-1 description: Computer Architecture Simulator supported: no name: ENV/DEVELOPMENT/GCC-4.1 description: The GNU C compiler, version 4.1 supported: no name: APPS/BIO/WEKA-3.4.10 description: WEKA Machine Learning Software supported: yes name: ENV/FORTRAN/G95 description: The GNU Fortran 95 compiler supported: no name: APPS/BIO/TFBS-0.5.0 description: TFBS supported: no name: ENV/DEVELOPMENT/G++-4.1 description: The GNU C++ compiler, version 4.1 supported: no name: APPS/BIO/LAGAN-1.2 description: LAGAN supported: yes name: APPS/GRAPH/POVRAY-3.6 description: The Persistence of Vision Raytracer supported: no name: ENV/PARALLEL-1.0 description: PARALLEL environment supported: no name: ENV/JAVA/JRE-1.5.0 description: Sun Java Runtime Environment of version 1.5 supported: no name: ENV/JAVA/SDK-1.4.2 description: Java2 Software Development Kit supported: no name: ENV/MPI description: MPI environments branch supported: no name: ENV/LOCALDISK-0000 description: Advertises the availability of local scratch disk on compute nodes. The number trailing RE name (here 0000) is the available disk space in MegaBytes. supported: no name: APPS/MATH/ELMER-5.0.2 description: Finite Element Software for Multiphysical Problems supported: no name: APPS/STATISTICS/R-2.3.1 description: R supported: no name: ENV/JAVA/JRE-1.4.2 description: Java2 Runtime Environment supported: no name: APPS/BIO/JASPAR-CORE-1.0 description: JASPAR-CORE supported: yes name: APPS/BIO/WEKA-3.4.8A description: WEKA Machine Learning Software supported: yes name: APPS/BIO/AUTODOCK description: AutoDock is a suite of automated docking tools. It is designed to predict how small molecules, such as substrates or drug candidates, bind to a receptor of known 3D structure. supported: yes 0 Successfully retrieved information.
This is already impressive. But most of the above are just the regular runtime envivonments that one knows from the NorduGrid RTE page. Those are available as MetaPackages, but with no package implementing them. In currently empty sections above, the MetaPackages would be shown if they were already installed by some means. And there are no jobs currently being registered with any RTE. To only see the installable list of MetaPackages (supported or not), do
/usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list installable
See the janitor(8) man page for further options to display the status.
Performing the automated installation
The automated installation makes sense only when there is a job demanding the availability of the package. Those jobs then have a jobid to identify it and the Janitor uses that for the associated of runtime environments with a job. So, after the job has run, one can then delete the runtime environments that are of no further use. But one could also think of abstract "resons" or almost tangible "projects" that could be given a number, too. So, you would delete a RTE only after that project has ended or there no reason any more to keep it. The registration with a jobid or projectid works like follows:
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf register 0 APPS/BIO/AUTODOCK 1 Sucessfully initialized job.
Now, the Janitor-internal information system lists its system as INSTALLABLE
/usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list rtes | head -10 Locally installed runtime environments: Dynamically installed runtime environments: name: APPS/BIO/AUTODOCK state: INSTALLABLE lastused: 1267096878
There are no locally installed runtime environments, which would be the the classical runtime environments, but we now have a dynamically installed on registered locally that is not installed but 'installable', so the Janitor thinks because it has found a package. Now for the deployment:
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf deploy 0 1 Cannot provide requested RTEs: installation FAILED.
The state of the previously installed runtime environment has changed accordingly.
Dynamically installed runtime environments:
name: APPS/BIO/AUTODOCK state: FAILED lastused: 1267096878
The "lastused" date is the UNIX epoch, i.e. seconds since 1970. We still need to find out why this has not worked. Then the installation fails, the Janitor assumes that the job will be rejected by the grid manager and consequently does not register the job. The information about he failure remains persistent, so subsequent attempts will falter immediately and are announced as such by the information system.
Since there is no job assigned, the failed RTE also cannot be removed, i.e. it is the job that is removed (deregistered) and the RTE then goes (after some expiration time) with it. To remove the failed RTE, we first set its state to removal pending.
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf setstate "REMOVAL_PENDING" APPS/BIO/AUTODOCK 0 Changed state successfully.
It is still there
Dynamically installed runtime environments: name: APPS/BIO/AUTODOCK state: REMOVAL_PENDING lastused: 1267096878
Until we now perform a sweep
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep [Janitor::Janitor] 2010/02/25 13:17:46 INFO> Removing APPS/BIO/AUTODOCK 0 Sweeping is done.
It is gone now. The challenge now is to either indicate a proper URL to the catalog entry. If it was not your very own catalog entry but some public one then it should have working packages announced not as a local URL but such accessible via http.
Preparation of a dynamic Runtime Environment package
The dynamic runtime environments may come in different flavours. Currently supported are tarfiles with two subdirectories:
- data - all the bits one expects from the user's perspective
- config - scripts to set the PATH and other environment flags
- install - perform post-processing after the data directory is copied to its destination
- remove - executed just prior to the removal of the files from the data directory.
- runtime - executed for every single job, repeatedly for multiple stages of job execution
The install and remove scripts, the latter commonly empty, are unique to the dynamic runtime environments. The runtime scripts are also required for the classical ARC runtime environments.
Building a very basic dRTE
As a playground binary let us prepare a HelloWorld application that prints exactly that.
[ -d data ] || mkdir data cat <<EOHELLOWORLD > data/HelloWorld #/bin/sh echo 'Hello World!' EOHELLOWORLD chmod +x data/HelloWorld
and there is not much to do for the install files etc..
[ -d control ] || mkdir control echo '#!/bin/sh' > control/install echo 'echo install invoked as: $0 $*' >> control/install echo '#!/bin/sh' > control/remove echo 'echo "Nothing extra to be done for removal."' >> control/remove echo '#!/bin/sh' > control/runtime echo 'export PATH=$PATH:%BASEDIR%' >> control/runtime echo 'echo runtime invoked as: $0 $*' >> control/runtime
The BASEDIR string will be substituted with the final installation directory.
Then this needs to be packed:
GZIP=-9 tar czvf HelloWorldRTE.tar.gz control data
a be made known to the Catalog.
Extending the Catalog
The Catalog is an RDF file. This way we can talk about some object in a distributed manner, i.e. we are not depending on the tree like structure. The objects/subjects you phrase triplets over (with predicates) all have their unique identifies, called URIs. Those look like URLs but do not need to exist.
To extend the Catalog, we introduce the concept of a HelloWorld package
cat <<ADDTHISTOTHECATALOG >> /var/spool/nordugrid/janitor/catalog/knowarc.rdf
<kb:MetaPackage rdf:about="&kb;MetaPackage/demo/hello/world"
kb:homepage="http://wiki.nordugrid.org/index.php/Janitor/"
kb:lastupdated="2010-02-28"
kb:name="APPS/DEMO/HELLO/WORLD"
rdfs:label="APPS/DEMO/HELLO/WORLD">
<kb:description>A minimalistic dynamic Runtime Environment, featuring the exutable script HelloWorld.</kb:description>
<kb:instance rdf:resource="&kb;package/tar/helloworld-0.0"/>
</kb:MetaPackage>
ADDTHISTOTHECATALOG
and then inform about he availability of a new TarPackage.
cat <<ADDTHISTOTHECATALOG >> /var/spool/nordugrid/janitor/catalog/knowarc.rdf
<kb:TarPackage rdf:about="&kb;package/tar/helloworld-0.0"
kb:url="http://www.inb.uni-luebeck.de/~moeller/dre/HelloWorldRTE.tar.gz">
<rdfs:label>HelloWorld </rdfs:label>
<kb:basesystem rdf:resource="&kb;BaseSystem/debian/etch"/>
</kb:TarPackage>
ADDTHISTOTHECATALOG
It is a matter of the individual sites to accept or deny claims for a particular base system.
The package will then be listed as
name: APPS/DEMO/HELLO/WORLD description: A minimalistic dynamic Runtime Environment, featuring the exutable script HelloWorld. supported: yes
Registration and Deployment
This time, registration was already
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf register 1 APPS/DEMO/HELLO/WORLD janitor: Warning: running with group root. [Janitor::Janitor] 2010/02/28 02:01:13 INFO> 1: registering new job (APPS/DEMO/HELLO/WORLD) [Janitor::Janitor] 2010/02/28 02:01:13 INFO> 1: sucessfully registered job 1 Sucessfully initialized job.
So, registration was successful:
Toshiba:/home/moeller/t# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list Jobs: jobid: 1 created: 1267318873 age: 16 seconds state: PREPARED rte: APPS/DEMO/HELLO/WORLD
And so is the deployment:
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf deploy 1 [Janitor::Janitor] 2010/02/28 02:02:07 INFO> 1: use basesystem "debian::etch_tar" for dynamically installed rte [Janitor::Janitor] 2010/02/28 02:02:07 INFO> installing package http://www.inb.uni-luebeck.de/~moeller/dre/HelloWorldRTE.tar.gz (Catalog: [NONAME]) [Janitor::Janitor] 2010/02/28 02:02:08 INFO> 1: sucessfully initialized job 0 Sucessfully initialized job.
Some closer inspection:
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list janitor: Warning: running as user root. janitor: Warning: running with group root. Jobs: jobid: 1 created: 1267318873 age: 6 minutes state: INITIALIZED rte: APPS/DEMO/HELLO/WORLD Locally installed runtime environments: name: APPS/DEMO/HELLO/WORLD Dynamically installed runtime environments: name: APPS/DEMO/HELLO/WORLD state: INSTALLED_A lastused: 1267318928
The UNIX epoch to display the last usage seems improvable. And, we should learn about where it is installed, well, that we know since we know where grid runtime environments are to be installed:
# grep runtimedir /usr/share/arc/janitor/conf/arc.conf runtimedir="/var/tmp/arc/runtime/config"
In that directory, the name of the app indicates the location of the configuration script, just like for the regular runtime environments
# find /var/tmp/arc/runtime/config /var/tmp/arc/runtime/config /var/tmp/arc/runtime/config/APPS /var/tmp/arc/runtime/config/APPS/DEMO /var/tmp/arc/runtime/config/APPS/DEMO/HELLO /var/tmp/arc/runtime/config/APPS/DEMO/HELLO/WORLD
But that does not necessarily indicate on the whereabouts of the installed runtime directory:
# grep installationdir /usr/share/arc/janitor/conf/arc.conf installationdir="/var/spool/nordugrid/janitor/runtime"
This way, the classical and dynamical runtime environments are clearly separated:
# find /var/spool/nordugrid/janitor/runtime /var/spool/nordugrid/janitor/runtime /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8 /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8/pkg /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8/pkg/HelloWorld /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8/install.log /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8/runtime /var/spool/nordugrid/janitor/runtime/Hell_TOHux7llB8/remove
To remove the dRE again, one dergisters the job
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf remove 1 [Janitor::Janitor] 2010/02/28 02:21:01 INFO> removing job 1 [Janitor::Janitor] 2010/02/28 02:21:01 INFO> 1: removed 0 Removed.
"0" is the exit status, here indicating no error.
The job is gone, but the RTE still present:
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf list Jobs: Locally installed runtime environments: name: APPS/DEMO/HELLO/WORLD Dynamically installed runtime environments: name: APPS/DEMO/HELLO/WORLD state: INSTALLED_A lastused: 1267320061
The job is removed with sweep,
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep [Janitor::Janitor] 2010/02/28 03:00:56 INFO> Removing APPS/DEMO/HELLO/WORLD [Janitor::Janitor] 2010/02/28 03:00:56 INFO> Removing HelloWorldRTE.tar.gz 0 Sweeping is done.
with sweep --force if one does not want to respect minimal lifetimes or other issues.
# /usr/lib/arc/janitor -c /usr/share/arc/janitor/conf/arc.conf sweep --force 0 Sweeping is done.
Working with Packages from Linux distributions
Retrieval of packages
One can get Packages from Linux distributions directly from the ftp or http servers (mirrors) of the distribution. Alternatively, particularly if interested in experimenting also locally with the software, one may prefer the local installations, first:
# apt-get install clustalw autodock autogrid boxshade [....]
and then save the .debs in a local folder
# mkdir debs # cp /var/cache/apt/archives/*.deb debs/
before removing them again
# apt-get clean
This is somehow similar for all the distributions.
deb2rte
One could think of various ways to integrate Debian or another distribution with grid computing:
- chroot environments (like Debian build daemons)
- virtual machines to be configured at runtime
- ... ?
We have played with the above, but beyond reporting that it works, we have not investigated that further. For this introductory round, we present the conversion of Debian packages into ARC-typical runtime environments.
editing not yet complete -- Steffen
Execution
This shall obviously happen prior to the removal of the runtime enironment. The exact name of the installation directory is not known in advance and not guaranteed to remain the same between job invocations. The executing scripts completely depends on environment variables to be set by the runtime script.
$ cat /var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/runtime #!/bin/sh export PATH=$PATH:/var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg echo runtime invoked as: $0 $*
The pkg directory is our former data directory:
$ ls /var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg HelloWorld
and when exuting the runtime script (performed automatically by the grid backend scripts), it indeed works:
$ export PATH=$PATH:/var/spool/nordugrid/janitor/runtime/Hell_ZJdVAIjuVG/pkg $ HelloWorld Hello World!
Integration with A-REX and ISIS
Verification of this paragraph is still pending: If the configuration files of the Janitor are performed correctly, then (upon a manual invocation of the Janitor) there will be no difference to the classical runtime environments. When ARC is compiled with the option to support the Janitor, then this will be performed with no further configuration.
The available packages are also forwarded to the grid information system. This goes beyond what was available for classical runtime environments, i.e. packages that are available but not yet installed.
