EMI Development

This page summarizes the ARC development plans within the EMI project. It was used as a workarea for the preparation of the EMI developoment plan of ARC product teams.

Contributors: Aleksandr, Daniel, Martin Skou, Zsombor, Jozef, Balazs, Mattias, Anders, Oxana, ...

• Notes from the June 18 2010 ARC EMI planning meeting

ARC CE

components: A-REX, Grid Manager, gridftp jobplugin interface, CE-Cache, CE-staging, LRMS modules, Janitor, Jura accounting hook

Proactive maintenance

• Hardening of the following components: A-REX, Janitor, CE-Cache, LRMS modules

Harmonization

• phasing out the gridftp jubplugin interface and replacing it with the WS inteface of A-REX component
• phasing out the classic Grid Manager and replacing it with a new one part of the A-REX component
• server-side support of the to-be-agreed AGU execution service interface
  • support consolidated Job Description Language (AGU-JSDL)
  • support consolidated glue2 schema
  • support the "AGU-BES"
• Remove dependency on the Globus toolkit by making globus dependent part plugable.

Evolution

• Implementing a new data staging system for the CE
• parallel support
• virtualization
• monitoring sensor
• remote service management
• accounting hooks
• interactive jobs support
• job request validation on the server-side
• server-side for hold/resume

ARC Compute Clients

components: ng* (list of commands) and arc* (list of commands) compute CLIs, libarcclient

Proactive maintainence

• Maintaining the arc* compute clients and the libarcclient
• Resource discovery and brokering speed-up.
• job submission speed up
• Harden the job description translators (xrsl, jdl, jsdl)
• Harden ACCs (midleware-specific plugins)

Harmonization

• support the to-be-agreed AGU execution service interface on the client-side
  • support consolidated Job Description Language (AGU-JSDL)
  • support consolidated glue2 schema
  • support the "AGU-BES"
• propose the libarcclient library as a general purpose EMI client-side "compute and info discovery" library

Evolution

• adjust libracclient to satisfy the needs of the ArcJobTool GUI, support the developer team
• adjust libarcclient to satisfy the needs of the P-GRADE portal, support the developer team
• adjust libarcclient to satisfy the needs of the Ganga tool, support the developer team
• Extend the support for job migration
• Extend the support for hold/resume (archold) (requires server side support)
• Advancing and extending the functionality of the data broker module
• Extend the resource discovery module
• Run job interactively (arcint) (requires server side support)
• Clever queuing
• develop a plugin to be able to submit jobs to localhost as an "execution server"
• develop a plugin to able to submit jobs directly to LRMS (no middleware installed on cluster)
• develop a plugin to be able to submit jobs to Clouds
• add, extend support of the client-side library on non-linux platforms (e.g. pyton, java on windows)

ARC Data libraries

components: ng* data cli, arc* data cli, libarcdata2 library, DMCs

Proactive maintenance

• maintaining the arc* data clients and the libarcdata2, fixing bugs

Harmonization

• phase out the ng* data clients and underlying libraries, encourage users to move over to the arc* cli and new libs.The ng* cli will be supported as long as it is used
• propose the libarcdata2 library as a general purpose client-side data library
• support the to-be-agreed SRM flavour.
• implement the necessary client-side changes related to the to-be-made agreement around GSI.
• implement the necessary client-side changes related to the to-be-made decision around Catalogue re-engineering

Evolution

• investigate the possibilities and needs to support nfs4.1 in the data libs
• investigate cloud storage area support within the client-side data library
• add support for xroot

ARC Classic SE

component: gridftp server

• proactive maintenance: none, only ordinary support activity as long as component is used
• harmonization: the component is to be phased out from the EMI software stack because there is no known need for standalone gridftp servers, the currently deployed gridftp servers are backends of larger SRM systems.
• evolution: none

ARC LDAP-based Infosys

components: Classsic Infoserver (localldap), Classic Infoindex (EGIIS), Grid Monitor, infoproviders

• proactive maintenance:
  • local ldap server: support the latest BDII version
  • EGIIS:
  • Grid Monitor: none
  • infoproviders: none

• harmonization:
  • support for glue2 schema all over the components
  • integrate with site info services (site-level info aggregation is missing in ARC)

• evolution:
  • add support for publishing information about virtualized and elastic systems
  • improve support for parallel environments and multicore resources in the infoproviders
  • A monitor is needed that supports the information system of WS-based ARC services in addition to the current LDAP.

ARC security utils

components: update-crls, nordugridmap, arcproxy

• proactive maintenance: none
• harmonization:
  • phase out update-crls from the ARC deployments and replace it with the fetch-crl
  • nordugridmap as a base for a common gridmap file generator utility.
  • arcproxy as the general purpose EMI proxy creation and manipulation tool
• evolution:
  • arcproxy: ARGUS integration, VOMS-SAML integration, improved myproxy support
  • mapfile generator: ARGUS integration

ARC Container

products: HED with all of its libraries such as LIDI, security framework, language bindings,

• proactive maintenance: performance enhancements, interface cleanup, massive cleaning in the security framework code, bringing java up-to-sync
  • complete the sysadmin friendly configuration layer
• harmonization: HED is a central component in ARC and can not be replaced by other container.

• evolution
  • information interface (LIDI)
    • implement support privacy (access control over published information)
    • implement information port/operations of an efficient information interface and a model
  • security framework:
    • integration with EMI-defined AAI solutions
  • language binding:
    • add, extend support of language binding libraries on non-linux platforms (e.g. pyton, java on windows)