This wiki is obsolete, see the NorduGrid web pages for up to date information.

NOX/Client tutorial

From NorduGrid
Jump to navigationJump to search

Client Tutorial

This tutorial will show the basic job configuration a and job handling via the Command Line Tool (CLI)

Help

In order to see all options of a command there several possible help calls:

><command> -h
><command> -?
><command> --help

On linux systems one van also use

>man <command>

Certificate

In order to access grid resources a user has to have a certificate. Every country has its own certificate issuing authority (CA) which defines specidifc procedures. Please consult The International Grid Trust Federation site in order to locate your country's CA.

For residents of Nordic countries (Denmark, Finland, Norway, Iceland and Sweden) one has to install the necessary Globus packages and the NorduGrid CA configuration package, and then run a certificate request command:

grid-cert-request -int -ca

filling in the information which is being asked for. The certificate request (usercert_request.pem file) has to be sent by e-mail to ca@nordugrid.org. More information about certificates is available here:

Instant CA

For testing one can acquire an temporary certificate from the KnowARC instant Certification Authority portal:

https://vls.grid.upjs.sk/CA/index

Fill out the form and follow the instructions in the readme file.

Note: You have to ask resource owners to install your instant CA credenials

arcproxy

A grid session is started by creating a proxy

>arcproxy

Check the proxy information:

>arcproxy -I

To create a voms proxy use option -S:

>arcproxy -S <vo_name>

This is only the simplest option, one can also specify group an role. See the help function for more information.

To submit jobs to the CREAM service, gLite, a GSI proxy has to be created: >arcproxy -O -S <vo_name>

The KnowARC test CREAM service can be used by members of the knowarc.eu virtual organization. Contact

Create a job

The new and standard compliant way of specifying jobs is via the Job Submission Description Language (JSDL)

Copy the definition below into a file, example testjob.jsdl.

<JobDefinition
 xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl"
 xmlns:posix="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix"
 >
 <JobDescription>
   <JobIdentification>
     <JobName>test_1 job </JobName>
   </JobIdentification>
   <Application>
     <posix:POSIXApplication>
       <posix:Executable>/bin/sh</posix:Executable>
       <posix:Argument>test.sh</posix:Argument>
       <posix:Argument>Testing</posix:Argument>
       <posix:Output>out.txt</posix:Output>
       <posix:Error>err.txt</posix:Error>
     </posix:POSIXApplication>
   </Application>
   <DataStaging>
     <FileName>test.sh</FileName>
     <DeleteOnTermination>false</DeleteOnTermination>
     <Source><URI>http://www.fys.uio.no/~katarzp/test/test_1.sh</URI></Source>
   </DataStaging>
   <DataStaging>
     <FileName>out.txt</FileName>
     <DeleteOnTermination>false</DeleteOnTermination>
   </DataStaging>
   <DataStaging>
     <FileName>err.txt</FileName>
     <DeleteOnTermination>false</DeleteOnTermination>
   </DataStaging>
 </JobDescription>
</JobDefinition>

xrsl - still available

All though the JSDL is the mother tongue of the new ARC, one can still use the classical way of specifying a jobs in xrsl. The job specification will be translated before submission.

This is a simple job which downloads a program from a web location, compiles and runs it. The output is a list of prime numbers. Copy the code into a file, for example testjob.xrsl.

&("executable" = "run.sh" )
("arguments" = "2" )
("inputfiles" = ("run.sh" "http://www.fys.uio.no/~katarzp/test/run_1.sh" ) 
("Makefile" "http://www.fys.uio.no/~katarzp/test/Makefile" ) 
("prime.cpp" "http://www.fys.uio.no/~katarzp/test/prime.cpp" ) )
("stderr" = "primenumbers" )("outputfiles" = ("primenumbers" "" ))
("jobname" = "ARC primenumber test" )
("stdout" = "stdout" )
("gmlog" = "gmlog" )
("CPUTime" = "8" )

Submit job - arcsub

As default submission without any arguments will use the default services

>arcsub testjob.jsdl

A job can be directed to a certain computing service either using an alias or the

>arcsub -c ARC1:http://knowarc1.grid.niif.hu:50000/arex testjob.jsdl
>arcsub -c areax1 testjob.jsdl

for xrsl and ARC0 services:

>arcsub -c arc0 testjobs.xrsl
>arcsub -i ARC0:ldap://index1.nordugrid.org:2135/Mds-Vo-name=NorduGrid,o=grid testjob.xrsl

Once a job is submitted it is assigned a unique job ID for example http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Check status - arcstat

To see the status of all jobs use option -a of for a particular job, the job ID

> arcstst -a
> arcstat http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

To get more detailed listing use option -l

> arcstat -l http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Read the output - arccat

As the jobs is executing one can ant any point have a look at the log files

> arccat http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Shows the standard output.

> arccat -l http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Shows the grid manager log file.

Get the results - arcget

Once the job is finished one can download the results using arcget

> arcget http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

As directory named after the session directory will be crated and it will contain all output files 1130412584325021155367992. Only files specified as output (xrsl) or in JSDL exuiped with the tag <DeleteOnTermination>false</DeleteOnTermination> will be kept for download. All other files only required for the execution of the job will be remove from the session directory when the job finishes.

Kill job - arckill

A job can be killed using the arckill command:

> arckill http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Jobs at a certain cluster can be killed using the -c/--cluster option:

> arckill -c arex1

Jobs with a certain status can be killed using the -s/--status option:

> arckill -s PREPARING

By default a job that is killed will be removed from the cluster. If one wants to download the output, for example to study error messages, one must use the -k/--keep option.

Clean jobs - arcclean

If jobs are not downloaded they remain for some time at the cluster. The jobs and it's session directory can be removed using the arcclean command.

Examples: clean single job, jobs with status FINISHED, jobs at cluster with alias arex1:

> arcclean http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992
> arcclean -s FINISHED
> arcclean -c arex1

Data handling

ARC offers several different ways of datanadling and supports a number of protocols. This section show only some very basic examples.

List files - arcls

One can at any moment check the content of the session directory of a job:

> arcls http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992

Or to list the content of some storage location:

>arcls srm://grid.uio.no:8446/srm/managerv2?SFN=/johndoe/log2

detailed listing is obtained with the -l/--long option.

Copy files - arccp

Files stored on the grid can of course be copied to a local file system using the arccp command:

> arccp http://knowarc1.grid.niif.hu:50000/arex/1130412584325021155367992/remote.file local.file
>arcls srm://grid.uio.no:8446/srm/managerv2?SFN=/johndoe/log2 log2.local

ARC offers also a lightweight distributed storage system - Chelonia. For a full manual see ARC Client manual

Monitor

A short while after the jobs are submitted they will appear in the monitor.

Test clusters monitor

  • The full NorduGrid production monitor

NorduGrid monitor