Data Staging/API

This page describes how to use the data staging library in a third party application. The library is documented as part of the ARC SDK documentation. As the library is written in C++, the C++ bindings are by far the easiest to use and provide the most functionality. Java bindings are also fully functional but require certain tweaks. Python bindings have also been tested, but lack certain features.

The datastaging library libarcdatastaging implements the lower two thirds of the data staging architecture shown here, i.e. the Scheduler and Delivery layers. To use the library an application must implement a Generator, to enter DTRs into the system and receive them back.

To run an application requires several packages (and their dependencies) to be installed. See the NorduGrid downloads page for how to get these packages.

• The core arc packages:
  • nordugrid-arc
  • nordugrid-arc-hed
  • nordugrid-arc-arex (required for ARC release 12.05 only, due to a packaging bug fixed in 12.05 update 1 and subsequent releases)
• Plugins for protocol support. Each package is optional and only needs to be installed if the corresponding protocol needs to be supported.
  • For ARC < 13.02:
    • nordugrid-arc-plugins-needed (plugins for protocols HTTP(s), SRM and file)
    • nordugrid-arc-plugins-globus (plugins for protocols GridFTP, LFC, RLS and xrootd)
  • For ARC >= 13.02:
    • nordugrid-arc-plugins-needed (plugins for protocols HTTP(s), SRM and file)
    • nordugrid-arc-plugins-globus (plugins for protocols GridFTP and LFC)
    • nordugrid-arc-plugins-xrootd (plugins for protocol xrootd)
    • nordugrid-arc-plugins-gfal (plugins for protocols RFIO and DCAP)

Other packages are required, depending on the language.

C++

A complete example of a basic Generator in C++ can be found in the ARC SDK documentation. The package nordugrid-arc-devel is required to provide the header files.

This example is very simple - it just copies a local file to another local file - and does not include any configuration of Scheduler or DTR options. A much more complex C++ example can be seen in the Generator for A-REX.

Java

A working example of a Generator in Java is available from the NorduGrid svn repository. This code requires the nordugrid-arc-java package.

Python

A working example of a Generator in Python is also available from the NorduGrid svn repository. This code requires the nordugrid-arc-python package.

As stated in the comments in the code, the main missing piece in Python is the ability to get callbacks to the Generator when the DTR has finished, so the Generator simply polls the state of the DTR until it finishes.