time: 10/9 2015 10:00 - 17:00
Location: NBI UCPH - Lille frokoststue (across from the kantine, Blegdamsvej 19)
10:00 - 10:30 localtransfer
10:30 - 12:00 common lrms calls between infosys and jobcontrol backends
13:00 - 15:00 re-integrating python-lrms
15:30 - 17:00 API and Ze Future
localtransfer should be removed from the backends, and the old uploader and downloader should be removed. We should keep the option of reimplementing the feature if there are requests. The option should be kept in arc.conf and the backends should log a suitable warning if it is turned on.
CUS - remove from the backends, documentation, add warning etc.
AW - remove uploaders and downloaders from distribution
Oxana - register as known issue
common lrms commands
We need a process that uses batch system command to fill in a table with full job and queue information. A full batch system state. This should be the full information needed for scan and the information system. When this job is done, it moves the information to a common file used by both backends and informationsystem.
We are targeting SLURM and CONDOR.
List all information needed for infosystem and scan - Florido, Chrulle
Decide on a "format" for common file. - Florido, Chrulle
Get the information from lrms - Florido, Chrulle
Add ssh option to gridmanager configuration section - Martin
Add new backend options to a-rex, conf and move over backends from branch - Christian?
Create separate package for python backends containing the inline python - AW
Finish overview table. Exactly who is writing/reading what in the control dir - Christian
Investigate whether localtransfer is used by any sites - Christian
It does not seem to be in use anywhere. The following sites are not using it: UCPH, IJS, Arnes, HPC2N, UiO, CSC, Glasgow, Bristol, RAL, brunel, Cern, UA, UNIBE No sites have reported it in use
- Create a full overview of the "API" i.e. all information flowing between arc and the backends - Christian, Florido
|joboption_lrms||Which lrms is configured||backends set this||should not have joboption prefix. It is not a joboption.|
- Investigate what lrms functions are called and with which options -
condor_rm <joboption_jobid>%.`hostname -f`
condor_q -constraint "NiceUser == False" -format "ClusterId = %V\n" ClusterId -format "ProcId = %V\n" ProcId -format "JobStatus = %V\n" JobStatus -format "CurrentHosts = %V\n" CurrentHosts -format "LastRemoteHost = %V\n" LastRemoteHost -format "RemoteHost = %V\n" RemoteHost -format "ImageSize = %V\n" ImageSize -format "RemoteWallClockTime = %V\n" RemoteWallClockTime -format "RemoteUserCpu = %V\n" RemoteUserCpu -format "RemoteSysCpu = %V\n" RemoteSysCpu -format "JobTimeLimit = %V\n" JobTimeLimit -format "JobCpuLimit = %V\n\n" JobCpuLimit
qconf -sconf global
scontrol show config
wsclient -e <endpoint URL> -m status -j <jobid>
code in Sysinfo.pm
- Investigate benchmarking tools and benchmark backends - ?
- for PERL, NYTProf enables the interpreter to aggregate and report data about performance without polluting the code. Synopsys from http://search.cpan.org/~timb/Devel-NYTProf-6.02/lib/Devel/NYTProf.pm :
# profile code and write database to ./nytprof.out perl -d:NYTProf some_perl.pl # convert database into a set of html files, e.g., ./nytprof/index.html # and open a web browser on the nytprof/index.html file nytprofhtml --open # or into comma separated files, e.g., ./nytprof/*.csv nytprofcsv
- Investigate ssh in HED - Silje,Martin
inline python - Jon
Available in EPEL and Debian (turned out a gregor herrmann added it to debian in July...)