This wiki is obsolete, see the NorduGrid web pages for up to date information.

Data Staging/URL Options

From NorduGrid
Jump to navigationJump to search

Solution to URL options problem

URL options are an ARC extension to standard URLs and are used to specify ARC-specific options to input and output files, for example whether the file should be cached or which type of checksum to calculate. The syntax is


where option has the syntax name=value. Furthermore, indexing service URLs can have the URL of a physical replica embedded within them:


This syntax can become complicated and confusing for users. URLs for input and output data are used in three different areas of arc, and with each is described with the solution currently implemented or planned to be implemented:

  • Command line tools such as arccp
    • Use command line options, for example
arccp --location=srm:// --dest-option checksum=md5 myfile lfc://
  • Job description - XRSL
    • Add attributes to each input or output file, for example
(inputfiles = ("file1" "lfc://" "cache=yes"))
(outputfiles = ("output" "lfc://" "location=srm://"))
  • Job description - JSDL
    • Use sub-nodes of Source and Target
  • Files used by the downloader and uploader ( and
    • Add key-value attribute pairs after the local and remote filenames, for example
file1 lfc:// cache=yes
  • As these files are internal to ARC and rarely seen by users, they can retain the old syntax for now
    • This means advanced users using dynamic output files still have to know the old syntax