This wiki is obsolete, see the NorduGrid web pages for up to date information.

ARC Compute Clients/Secure functionality over transition

From NorduGrid
Jump to navigationJump to search

The following tasks should be carried out in order to preserve functionality and improve usability between the classic and WS compute clients.

Tasks with no assignee

  • arcstat can only query a job if its description is in the jobs.xml (or in some other file given by -j) - ngstat can query any jobs which ID is given to it, even if it is not in .ngjobs or any joblist. arcstat should be able to get job information from a job ID, which is not in the job list.
- Assigned to:
  • Add follow feature in arccat (-f option in ngcat)
- Assigned to:
  • Migration documentation: A section on client migration in each WS client man page, if applicable.
    Document that:
    • arc* cannot get the list of clusters from a file (-C in ng*)
    • arc* cannot get the list of GIISes from a file (-G in ng*)
    • Timeout in WS arc is for all communication steps. The timeout is applied on each step. Connect, close, send info, etc.
    • Anonymous queueries are used in WS client, and that it is not possible to use gsi-gssapi binds (-x and -X flags in the ng-client).
    • ngsync always truncates the .ngjobs file, then fill it up with the jobs it found in the specified clusters, arcsync by default merges the existing jobs with the found ones, and only truncates, if -T is given.
      Document new behaviour/feature.
    • There is no queue info (-q) in arcstat as compared to ngstat. In WS client the functionality of ngstat has been split into arcstat and arcinfo. The functionality of arcinfo corresponds to that of "ngstat -q".
    • In WS client long options must be prefixed with double dashes while in classic client only a single must be used.
    • ngresub -k specifies the target clusters - arcresub -k is for keep, and -q is for specifying the target clusters
    • ngrenew refuses to renew the proxy if it's not older then the local proxy - arcrenew renews it in this case also. Describe the difference.
- Assigned to: Low priority.
  • The hostname XRSL attribute added by ng(re)sub and arc(re)sub differs. hostname value of ngsub additionally contains domain part, where as arc(re)sub does not.
- Assigned to: Low priority.
  • Improve handling of short URLs
    • Only one error message should be printed when a endpoint fails to respond.
- Assigned to:

Tasks with assignee

  • JobDescription class and parsers improvements
    • Bring the xRSL parser of WS ARC (XRSLParser) in sync with the parser in Classic ARC. I.e. they should have the same behaviour.
    • ng(re)sub only allows unknown attributes with -U, arc(re)sub ignores them.
- Assigned to: Martin
  • Create arctest man page
- Assigned to: Ivan
  • Correct the behaviour of blacklisted services/clusters in WS ARC. Blacklisting of services (using -<cluster> notation, and blacklisting in client conf) should be tested and the behaviour should be fixed if not working properly. Currently blacklisting does not work if only the -<cluster> notation is used on the command line, without specifying any selected services. E.g.:
$ arcsub -c-ARC0:... -c-ARC1:... myjob.xrsl
In that case the default services from the client configuration should be used and the specified CEs should not be considered.
- Assigned to: Martin
  • Proxy issues:
    • Fix bug 1783 - arcstat asks for passphrase even if proxy exists
- Assigned to: Martin
  • If proxy expired, ngsub prints "The proxy has expired". arc(re)sub prints: "Submission to gsiftp://pgs02.grid.upjs.sk:2811/jobs failed, trying next target Job submission failed, no more possible targets". The message should instead inform that the proxy has expired.
- Assigned to: Salman
  • Filter submission to CE according to authorisation (A-REX and multiple queues)
- Assigned to: Martin
  • Remove dependency on job list file from library and output printed by WS client which need to be corrected:
    • arcget should print summary ("Jobs processed: X, successfuly downloaded: Y"), and where data have been stored (e.g. "Results stored at /tmp/outputs/284991296140979389378995")
    • arckill should print summary similar to ngkill
    • arcclean should print summary similar to ngclean
    • arcresume should print summary similar to ngresume
    • arcrenew should print summary similar to ngrenew
    • arccat should only output the file content like ngcat (and cat)
    • arcsub should print summary in case of multiple job submission similar to ngsub
- Assigned to: Martin

Completed tasks

  • Improve handling of short URLs
    • The UNICORE and BES TargetRetriver classes need to be modified. See mail by Zsombor (18/01/11).
- Assigned to: Mattias (fixed by Aleksandr)
  • Investigate what aspects of the ngtest command which can be ported, and port these to WS ARC creating the arctest command.
    • The existing functionality will be implemented (testing features against ARC0 clusters)
    • As much part of arcsub will be reused as possible
    • The usability of arctest will as similar to ngtest as possible
- Assigned to: Ivan
  • Migration tool/script able to migrate configuration files from classic to WS ARC client. It should done in bash.
    • Convert giis conf to UserConf. The following giis conf locations should be considered:
1. giis options in client section in $HOME/.arc/client.conf (giis = url)
2. giis options in common section in $HOME/.arc/client.conf
3. $HOME/.nggiislist (one GIIS per line, ARC ≤ 0.5.48)
4. giis options in client section in $ARC_LOCATION/etc/arc.conf
5. giis options in common section in $ARC_LOCATION/etc/arc.conf
6. $ARC_LOCATION/etc/giislist (one GIIS per line)
7. giis options in client section in /etc/arc.conf
8. giis options in common section in /etc/arc.conf
9. /etc/giislist (one GIIS per line)
  • Convert old style alias to new style.
  • Convert jobs in .ngjobs to jobs.xml.
- Assigned to: Zsombor
  • Documentation to be improved:
    • The documentation about the ngstat/arcstat does logical OR between -a, -j, -c, -s and list of jobids is not right: while multiple -c clusters and list of jobids have logical OR, but -s does filtering on the results (kind of logical AND), and -a is ignored completely when anything else is given; contrary to the documentation, -s does not need -a or -c in order to work. So the example: "For example, ngstat -s FINISHED -c my.grid.org <jobid> will return information about all finished jobs on the Grid, plus about all jobs (in any state) on the cluster my.grid.org, plus about the job <jobid>." should be "will return information about only the finished jobs on the cluster my.grid.org and among the listed jobid." Same for arcstat.
- Assigned to: Zsombor
  • Add attribute to UserConf to be able to specify default download directory for arcget
- Assigned to: Salman
  • Improve the description of arc* -j, it is confusing.
- Assigned to: Zsombor
  • arcget -J option need to be documented in the man pages and in the Users manual.
- Assigned to: Zsombor (fixed by Gabor)
  • joblist feature in job management commands in ng clients
    arc client job managements commands needs the joblist file in order to manage jobs, ng commands only need jobid. In ng client it is possible to specify a file with a list of IDs to manage using -i, no such options for arc client commands.
    The -i used to specify index services will be changed to -g. -i will instead be used for file containing list of job ids.
- Assigned to: Zsombor
  • Implement the -o option from the ng-client (joblist files: -o in ng(re)subs puts the new job ID into the specified file and also into ~/.ngjobs; -j in arc(re)sub puts the jobdescription into the specified file only, and not into ~/.arc/jobs.xml.
    Mattias informs that -o and .ngjobs files does not have same format, are not identical and are not used for the exact same purpose (bug exist on the subject).)
- Assigned to : Zsombor
  • arcget must skip downloading if download directory already exist
- Assigned to: Gabor
  • Compare command line arguments between the two versions, note and describe the differences. It might be that if there are differences between the two, WS ARC command line arguments should be changed to the behaviour of classic.
    Assigned to: Zsombor
  • Extend the JobDescription class and improve the XRSLParser to be able to support XRSL fully.
    • The JobDescription class should be extended to be able to hold a linked list of JobDescription objects,
      Assigned to: Martin
    • The xRLS parser should be fixes such that it makes use of the linked list in JobDescription class,
      Assigned to: Martin
    • The parsers should be improved such that they return a list of JobDescription objects when parsing a file. In general a file can contain multiple job descriptions.
      Fix bug 1870 - support for Multiple RSL in one file
      Assigned to: Martin
    • Bring xRSL reference manual up to date. It should document that certain attributes can only be specified once, and that for some attributes only certain comparison operators are allowed/considered. When bringing the manual up to date, the xRSL parser in Classic ARC should be used as reference.
      Assigned to: Martin
      Task completed by Oxana
  • Fix bug 1480 - Replace URLs in arcsub -c with short target names
    • This should be solved by first enabling short URLs where middleware has been specified. E.g. -c ARC0:example.com or -c ARC1:example.com/arex.
    • Then some clever guessing should be used. I.e. each specialised Submitter class should have a method which tries to figure out if the passed endpoint is supported by that submitter.
Assigned to: Mattias
  • Sensible default services should be set in the client configuration file. These services should most probably only be the services currently used as default in Classic ARC.
    Assigned to: Oxana
  • The output printed by the WS ARC cli should be similar/identical to that printed by the classic cli. I.e. every print to std::cout in the classic cli, should also be done in WS ARC.
Assigned to: Salman and Ivan
  • Fix bug 1876 - arcget does not work with subdirectories
Task completed by Gabor (bug was fixed)
  • Examine the possibility to support the dryrun attribute. Implementation properly needed at A-REX.
Assigned to: Martin (A-REX and arcsub now support dryrun)
  • Output printed by WS client which need to be corrected:
  • ngsync prints the IDs of all the jobs it founds, arcsync only prints the number of found jobs. arcsync must print IDs of all the jobs it finds.
- Assigned to: Salman
  • arcinfo should per default print queuename, and host name and long URL.
- Assigned to: Salman
  • Compare manual pages between the two versions and ensure that:
    • Everything described in classic man pages also are described in WS ARC ones.
    • If a functionality, described in classic man pages does not exist in WS ARC it should be noted that the functionality does not exist in WS ARC, and how it can be obtained.
    • Any relevant difference between the two sets of man pages should be noted, so it can be used to generate a transition document, describinsg what are the differences between the two versions.
Assigned to: Zsombor
Retrieved from "https://wiki.nbi.ku.dk/w/index.php?title=ARC_Compute_Clients/Secure_functionality_over_transition&oldid=3377"