This wiki is obsolete, see the NorduGrid web pages for up to date information.
ARC Compute Clients
From NorduGrid
Jump to navigationJump to search
This page concerns the "ARC Compute Clients" product in EMI.
The EMI project will last for 3 years, from May 2010 to April 2013, and will bring together the 3 major European middlewares ARC, gLite and UNICORE. In the project consolidation and harmonization of the 3 middlewares will be carried out, but there will also be room for proactive maintenance and evolution (development).
In this page and sub-pages, information about proactive maintenance, harmonization, evolution, testing and analysis related to the ARC Compute Clients product are gathered. This page contains a component list and a technical objective, work plan and open bugs table. If you have questions, comments or feedback please use the Discussion page.
Components
The product consist of all the computing related user command lines tools in ARC and the libarcclient library. Below these are listed:
- Pre-WS computing client
- ngsub - Submission
- ngresub - Resubmission
- ngstat - Job and resource querying
- ngget - Output retrieval
- ngcat - Output and log catenation
- ngkill - Kill job
- ngrenew - Credential renewal
- ngresume - Resume job from a failed state
- ngsync - Job list synchronization
- ngtest - Test client- or server setup
- ngclean - Job cleaning
- WS computing client
- arcsub - Submission
- arcresub - Resubmission
- arcmigrate - Migration
- arcstat - Job querying
- arcinfo - Resource querying
- arcget - Output retrieval
- arccat - Output and log catenation
- arckill - Kill job
- argrenew - Credential renewal
- arcresume - Resume job from a failed state
- arcsync - Job list synchronization
- arcclean - Job cleaning
- libarcclient
- Library used by the WS computing client
Milestones, Deliverables and Technical objectives
The following milestones, deliverables and technical objectives are related to the ARC Compute Clients product. The milestones and deliverables are collected from the EMI DoW, while technical objectives (TO) are taken from DNA1.3.1, more specifically from the table in section 5.2. A list of all the EMI milestones and their status can be found here, while information about deliverables is here.
| Name | Due month | State | Comments |
|---|---|---|---|
| DNA1.3.1 – Technical Development Plan | 2 | ONGOING | All input for the deliverable have been provided. Deliverable not delivered yet. |
| DJRA1.1.1 – Compute Area Work Plan and Status Report | 3 | ONGOING | All input for the deliverable have been provided. Deliverable not delivered yet. |
| MJRA1.13 - Agreement on common information exchange methods | 3 | UNCLEAR | |
| MJRA1.2 - Agreement on common job submission and management methods | 6 | ONGOING | F2F and phone meetings are being held to reach an agreement |
| Finalize the XML and LDAP rendering of the GLUE2 model (TO 26) | 9 | UNCLEAR | |
| DNA1.3.2 - Technical Development Plan | 11 | HOLD | Input for this deliverable should be provided approximately 1 month before due date. |
| DJRA1.1.2 - Compute Area Work Plan and Status Report | 12 | HOLD | Input for this deliverable should be provided approximately 1 month before due date. |
| MJRA1.14 - EMI Information exchange used in computing (TO 15) | 12 | HOLD | |
| MJRA1.3 - Successful implementation of the common job submission and management method (TO 14) | 18 | HOLD | Waiting for completion of MJRA1.2 |
| DNA1.3.3 - Technical Development Plan | 23 | HOLD | Input for this deliverable should be provided approximately 1 month before due date. |
| DJRA1.1.3 - Compute Area Work Plan and Status Report | 24 | HOLD | Input for this deliverable should be provided approximately 1 month before due date. |
| Consolidation and harmonization of compute area clients/APIs (TO 32) | 24 | ONGOING | Consist of improving libarcclient |
| DJRA1.1.4 - Compute Area Work Plan and Status Report | 36 | HOLD | Input for this deliverable should be provided approximately 1 month before due date. |
| Extend job definition languages, resource modeling schema (GLUE) and job management services to be able to request access to virtualized resource managers and appliances (TO 43) | 36 | UNCLEAR |
Work plan
Overview table
| Description | Priority | Assignee | Comments |
|---|---|---|---|
| #Improve technical documentation | Very high | Martin | Writing documentation of API. Also writing a guide on howto create a ACC module for libarcclient. |
| #Create a framework for web content management | Very high | ||
| #Detailed performance analysis | Normal | ||
| #Extend the resource discovery module | High | ||
| #Extend unit testing | Normal | ||
| #Create functional tests | High | ||
| #Revise READMEs and man pages | Normal | ||
| #Improve user documentation | Normal | ||
| #Write libarcclient article | Normal | Martin, Others? | |
| #Secure functionality over transition | Very high | All | |
| #ArcJobTool support | High | Martin | Support is ongoing. |
| #P-GRADE support | Normal | Contact need to be established. | |
| #Ganga support | Normal | Contact need to be established. | |
| #Improve data broker | High | Salman | |
| #Improve libarcclient language bindings | High | ||
| Bug fixing | High | ALL | Pick a bug! |
| #Extension to clouds | Low | This task might be appropriate for a student project. | |
| #Extension to remote LRMS | Low | This task might be appropriate for a student project. | |
| #Extension to localhost | Low | This task might be appropriate for a student project. | |
| #Interactive jobs | Low | This task might be appropriate for a student project. | |
| #Performance analysis of language bindings | Normal | ||
| #Extend support for job migration | Normal | ||
| #Support EMI ES | Very low | This task cannot be started before MJRA1.2 is reached. When reached priority will be highly increased. | |
| #Harden ACCs | Low | Gabor | Creating ACC unit tests. A framework for testing SOAP based client have been created. |
| #Harden the job description translators | Low | ||
| #Job submission and management speed up | Normal | ||
| #Resource discovery and brokering speed-up using cached info | Normal | ||
| #Resource discovery and brokering speed-up (ISIS) | Very low | Salman | This work can currently not be attributed to EMI since ISIS is not in EMI. |
Proactive maintenance
- Maintaining the arc* compute clients and the libarcclient
- Bugfixing, see the table in the #Open Bugs section.
- Resource discovery and brokering speed-up
- Salman worked on a solution where the ISIS hold a limited set of semi-statical attributes, however it has not been committed yet. Results were posted to the KnowARC mailing list. A continuation of this can probably not be attributed to EMI, since the ISIS is not part of EMI. Salman and Martin are working together on integrating this work.
- Another way this can be achieved is by caching semi-statically resource information at the client side. This might be work for a bachelor project or a autonomous project in the master studies.
- Job submission and management speed up
- Multi-job submission can be sped up by using a single connection when submitting multiple jobs to a single CE, see bug 808.
- Job management can probably be sped up when multiple jobs on a single CE should be managed. This should also be done by utilizing a single connection, see bug 1887.
- Carry out resource discovery, matching making and target ranking in parallel for each target. More information on this task can be found here.
- Harden the job description translators (xrsl, jdl, jsdl)
- Unit tests should be extended to make a ~100% coverage of the parsers.
- Go through the translators and verify that they behave properly, and that all attributes are supported.
Make job description translators plug-able.Done.- The JSDL parser should warn about unrecognized elements.
- Support for multiple XRSL in the same file.
- Support logical OR expressions in xrsl.
- Extend the JobDescription class to be able to support XRSL fully.
- Harden ACCs (midleware-specific plugins)
- Create functional tests, which can be used in a automated test frame work, like the one deployed by Marek in KnowARC.
- Create a unit tests which can test the ACC modules.
- Verify that every ACC is functional (develop tests), and resolve any issues.
- Improve Unicore support, capable of finding resources, but not yet submit jobs. Also investigate delegation procedure used in Unicore, it seems certificate and key are expected by the service as default.
- Extending arcsync to CREAM, see D2.2-1.
- Secure that all functionality in the pre-WS Compute client is transfered to the WS Compute client
Harmonization
- Support EMI Execution Service (ES) interface on the client-side
- The creation of the EMI ES specification is the subject of MJRA1.2 milestone. Work on implementing the specification cannot start before the milestone have been reached. Milestone MJRA1.3 concerns this implementation. More info on the specification is located here.
Evolution
- ArcJobTool support
- Adjust libracclient to satisfy the needs of the ArcJobTool GUI.
- Support the developer team.
- P-GRADE support
- Currently P-GRADE uses the CLI of the pre-WS client to interact with ARC resources. This presentation was presented at the PUCOWO 2010 conference.
- Contact P-GRADE developers to promote libarcclient and learn about their interest in libarcclient (Java bindings).
- Adjust libarcclient to satisfy the needs of the P-GRADE portal.
- Support the developer team.
- Ganga support
- Currently Ganga uses the CLI of the pre-WS client version 0.6 to interact with ARC resources. Ganga can probably also interact with ARC through Panda.
- Contact Ganga developers to promote libarcclient and learn about thier interest in libarcclient (Python bindings).
- Adjust libarcclient to satisfy the needs of the Ganga tool.
- Support the developer team.
- Extend support for job migration
- Currently only job residing on a A-REX CE can be migrated, and they can only be migrated to other A-REX CEs.
- Examine whether job migration from GM, CREAM or UNICORE can be supported. If not possible detailed information should be added as comments to the source code. If its possible, or limited possible a solution should be crafted.
- Same as above, but instead for job migration to GM, CREAM or UNICORE.
- Enabling migration to also get remote files from old cluster
- Extend the support for hold/resume (archold) (requires server side support)
- Currently there is no support for holding jobs in ARC. It is planned that the EMI ES will include hold operations. This task is thus part of #Support EMI ES.
- Advancing and extending the functionality of the data broker module
- A dedicated page concerning data broker module improvements is located here.
- Extend the resource discovery module
- All relevant GLUE2 information for the ARC1 plug-in should be communicated to and available on the client side.
- VO information should be fetched and used for match making.
- Run job interactively (arcint) (requires server side support)
- Might be work for a student project.
- Develop a plug-in to be able to submit jobs to localhost as an "execution server"
- This task should develop a plug-in which uses the local machine to run jobs. It should be investigated what features of libarcclient which can be supported. Grid job testing and utilizing the local machine as a execution service is motivating factors.
- Develop a plug-in to able to submit jobs directly to LRMS (no middleware installed on cluster)
- It should be investigated to what extend the libarcclient library is able to interact with remote LRMS. Remote LRMS should be understood as a LRMS run remotely without any middleware installed on it, and should be accessed through a dedicated connection. It might require lots of work supporting different connection protocols, on the contrary this task might benefit from the fact that the major LRMSs is already supported on the server side.
- Develop a plug-in to be able to submit jobs to Clouds
- It should be investigated to what extend the libarcclient library is able to interact with cloud computing.
- After investigation, a plug-in should be developed which uses the libarcclient API/interface. This task might be work for a bachelor project, an autonomous master study project or maybe even a master project.
- Improve libarcclient language bindings
- The language bindings should be tested in general. Unit tests for these should be created. Consider using junit and pyunit.
Java bindings on Windows and Mac should be fixed. See bugs.Fixed. There are currently no issues.
- ARC client on non-linux platforms
- Stand-alone client for Mac OS X should be created, see packaging repository.
- Language bindings should be packaged for Windows (assigned to Anders).
Performance
- Break down performance analysis of the WS computing client
- A detailed performance analysis should be carried out, which in depth analyses where computing time is spend, specifically it should be able to point to specific lines in the code. The analysis should be constructed so it can run in a automatic manner, and the results should be in a format applicable to a web page.
- Performance analysis of libarcclient with and without language bindings
- A performance analysis of the language bindings should be carried out, which compares performance of the libarcclient without language bindings. The analysis should be constructed so it can run in a automatic manner, and the results should be in a format applicable to a web page.
Testing
Note: Testing efforts should be coordinated with the SA2.5 task, more specifically the ARC partner UPJS (Marek Kocan). Marek also has testing experience from the KnowARC project. The ARC1/Testing page might contain valuable information. The EMI twiki page EmiSa2CertTestGuidelines contains EMI testing guidelines and test definitions.
- Extend unit testing
- Unit Testing of the libarcclient should be highly improved. It should be investigated which classes in libarcclient is applicable to unit testing, and unit tests should be created for them. The test coverage of the classes should be as high as possible, without changing the code of the class, see Test Coverage for more on test coverage.
- Create functional tests
- Functional tests should be created which can be run in a automated manner. It should be examined if the virtual testbed setup created by Daniel Johansson could be used for this purpose. Tests should be created for all the WS computing client commands, and the tests should cover all the common uses of command line options.
- It should be noted that the functionality of the arcresume and arcrenew was not completely ensured in the KnowARC project.
Documentation and Promotion
- Create a framework for web content management
- A framework for managing web content should be set up, in order to present documentation and such on the web. The content includes documentation, performance results, test coverage, code examples, turtorials, training material etc.
- Update READMEs files and man pages
- The READMEs and manual pages should be reviewed and updated. A lot of different READMEs exist (packaging READMEs).
- User documentation
- Review and improve user manual.
- Create a web page containing user documentation. Also a page for first time users should be created.
- Technical documentation
- Technical documentation already exists for the libarcclient. However it should be improved and extended. The documentation should preferably be written in a way which makes it applicable for a document (PDF) and a webpage.
- A web page containing technical documentation should be created. A general layout should be used, see #Create a general Web layout above.
- All classes in libarcclient should be attached with comments in doxygen style.
- A web page containing API documentation from the doxygen styled comments should be created, and they also explain how the class is used from Python and Java, if different from C++.
- Tutorials and code examples for using the libarcclient API should be created and presented in a web page. Python examples can be found here, and Java here.
- Write libarcclient article
- An article about the libarcclient should be written to promote its capabilities.
- The article should include a comparison of performance of the WS computing client against the pre-WS computing client. A performance analysis between the two have already been carried by Marek in the KnowARC project. That analysis should be investigated, and if not found sufficient for the article it should be extended.
Open Bugs
New and reopened
| ID | P | Version | Summary (5 tasks) ⇒ | Component | Assignee |
|---|---|---|---|---|---|
| 4213 | P3 | latest | Storage clients are not working with tokens | User Interface | Ievgen Sliusar |
| 4215 | P3 | unspecified | ARC 6 client can not submit to ARC 7 if -T arcrest interface option is not explisitly added | User Interface | Ievgen Sliusar |
| 4025 | P3 | 6.12.0 | arcsync produces duplicates of emies/arcrest jobs | User Interface | Martin Skou Andersen |
| 4218 | P3 | unspecified | ARC 7 client - still contains emies and gridftp logging info | User Interface | Ievgen Sliusar |
| 4033 | P3 | 6.13.0 | client.conf is ignored when using ARC6 options | User Interface | Martin Skou Andersen |
Blockers and criticals
no bugzilla tickets were found
General
| ID | P | Severity | Version | Summary (11 tasks) ⇒ | Component | Assignee |
|---|---|---|---|---|---|---|
| 4232 | P3 | major | 6.21.1 | VOMS test fails with openssl 3.4.0 | ARClib | Aleksandr Konstantinov |
| 3377 | P3 | minor | 13.11.1 | Unclear description of options -o, -i and -j in man pages of arcsub, acstat, arckill etc | User Interface | Martin Skou Andersen |
| 3362 | P4 | minor | 13.11.1 | arc* tools can not lower pre-configured verbosity level | User Interface | Martin Skou Andersen |
| 3661 | P3 | normal | latest | job submitted even if walltime exceeds the allowed value | User Interface | Martin Skou Andersen |
| 4226 | P3 | normal | 7.0.0a2 | ARC7 client suggests to install globus plugins talking to ARC6 server | User Interface | Ievgen Sliusar |
| 3361 | P3 | normal | 13.11.1 | Broken interpretation of executable location and variables | User Interface | Martin Skou Andersen |
| 3472 | P3 | normal | 15.03 | arctest does not print "plugin missing" error for org.nordugrid.gridftpjob interface submission | User Interface | Martin Skou Andersen |
| 3475 | P3 | normal | 15.03 | arcsub doesn't work with memory in job description | User Interface | Martin Skou Andersen |
| 4022 | P3 | normal | latest | python submission script does not exit, hangs in Arc::RunPump::Remove(Arc::Run*) | ARClib | Aleksandr Konstantinov |
| 3370 | P3 | normal | 13.11.1 | arcsync finds no jobs | User Interface | Martin Skou Andersen |
| 868 | P3 | trivial | 11.05 | Incorrect error message from arcsub when queue information is unavailable | ARClib | Florido Paganelli |
Enhancements
| ID | P | Version | Summary (9 tasks) ⇒ | Component | Assignee |
|---|---|---|---|---|---|
| 2627 | P3 | 11.05 | Increase job submission speed with arcsub | User Interface | Martin Skou Andersen |
| 3364 | P3 | 13.11.1 | Better message needed when job can not be found | User Interface | Martin Skou Andersen |
| 2673 | P3 | SVN | arcproxy should support batch query of validity | User Interface | weizhong qiang |
| 3388 | P3 | SVN | CE-Parallel Job Query for arcstat | User Interface | Martin Skou Andersen |
| 810 | P3 | 0.8.2.2 | Bad sites are repeatedly tried for multi-job job-submission | User Interface | Martin Skou Andersen |
| 2922 | P3 | 12.05 | All job management tools should report missing plugins | User Interface | Martin Skou Andersen |
| 3521 | P3 | 15.03 | Provide API to get the reason for failed matchmaking | User Interface | Martin Skou Andersen |
| 3335 | P3 | 13.11 | Client does not provide local input file size and checksum for EMI-ES | User Interface | Martin Skou Andersen |
| 1532 | P4 | 0.8.2.2 | add preferred SURL option | User Interface | David Cameron |
Feature requests
no bugzilla tickets were found
Minutes
Regular meetings in the ARC Compute Clients PT is taking place. Below you will find the minutes of these meetings:
KnowARC remnants
Command line tools status
| Command | ARC0 | ARC1 | CREAM | UNICORE |
|---|---|---|---|---|
| arcsub | Ok | Ok | Ok | Job submission but no staging |
| arcstat | Ok | Ok | Ok | Ok |
| arcinfo | Ok | Ok | Ok | Ok |
| arcget | Ok | Ok | Ok | No |
| arcclean | Ok | Ok | Ok | No |
| arckill | Ok | Ok | Ok | Ok |
| arccat | Ok | Ok | No | No |
| arcresub | Ok | Ok | Ok | No |
| arcmigrate | No | Ok | No | No |
| arcrenew | Ok | No | No | No |
| arcsync | Ok | Ok | No | No |
| arcresume | Ok | No | No | No |
| arctest | No | No | No | No |
A list of all the command line options used in the client commands can be found at ARC_Compute_Clients/Command_line_options.
Documents
- libarcclient F2F meeting in Lund (24-25 February 2009): Minutes
- libarcclient user documentation (user manual): arcclient.pdf
- libarcclient technical documentation: client_technical.pdf
- Mapping into ExecutionTarget.h: ExecutionTargetMapping.ods
- Mapping into Job.h: JobMapping.ods
- Internal Job Description: JobInnerReprensentation.ods
- Status mapping between the different middlewares: JobStatusMapping.ods
- Common Information Service (CIS) - UNICORE6: CIS-guide.pdf
- Unicore infosys: http://unicore.svn.sourceforge.net/viewvc/unicore/CommonInformationService/
- Interesting JSDL parameter sweeping: http://www.ogf.org/documents/GFD.149.pdf
- Unicore UCC client (this Unicore OGSA-BES client can work without Unicore registry): http://www.knowarc.eu/download/ucc-1.2.0.-do-not-need-registry.tgz
