This wiki is obsolete, see the NorduGrid web pages for up to date information.

Testing/ARC-Clients emi1 rc1

From NorduGrid
Jump to navigationJump to search

Test report

EMI_Test_Report_ARC-Clients_EMI1_RC1.pdf


Component Description

The ARC-Clients components consist of a set of commands necessary for job management, proxy manipulation and data handling. Also, the underlying libraries are included. The compute cli support different job description languages (XRSL, JSDL, JDL), different CE flavours (Classic ARC, A-REX, gLite Cream CE). The data cli handles different storage elements. The arcproxy serves as a general purpose proxy manipulation utility.

Code analysis

  • Tester: Marek
  • due: for every RC
  • Sloccount:
cpp:         136167 (74.26%)
perl:         18590 (10.14%)
php:          13007 (7.09%)
python:       10545 (5.75%)
sh:            4199 (2.29%)
ansic:          798 (0.44%)
java:            63 (0.03%)

Total Physical Source Lines of Code (SLOC) = 183,369

The complete results of SLOC count entire ARC code can be found here

______________________________________________________________________

  • CCCC metrics:
Metric Tag Overall Per Module
Number of modules NOM 890
Lines of Code LOC 106028 119.133
McCabe's Cyclomatic Number MVG 30720 34.517
Lines of Comment COM 12674 14.240
LOC/COM L_C 8.366
MVG/COM M_C 2.424
Information Flow measure ( inclusive ) IF4 618184 694.589
Information Flow measure ( visible ) IF4v 618047 694.435
Information Flow measure ( concrete ) IF4c 3341 3.754
Lines of Code rejected by parser REJ 3197

The complete results for CCCC for entire ARC code can be found here

Later the results will be split between components.

Unit tests

  • Tester: Anders
  • due: for every RC

Results of unit test code coverage for entire ARC code can be found here:

http://hep.nbi.dk/~waananen/nordugrid/20781/

The text description of results will come later.

Later the results will be split between components.

Regression tests

  • Tester: Marek
  • due: for every RC

initially the verification of resolved critical and blocker bugs

R1

R2

R3

R4

R5

R6

Deployment tests

D1: clean installation

  • tester: Marek
  • due: for every RC

D2: upgrade installation

  • tester: Marek
  • due: postponed

Functionality tests

F1: direct job submission

  • Tester: Marek, Jozef
  • due: 25 March

job submission to a pre-selected resource

  • ngsub
  • arcsub
  • test pre-WS, WS, CREAM, Unicore

F2: Job migration

  • Tester: Marek, Jozef
  • due: 25 March
  • arcmigrate

F3: job and resource info querying

  • Tester: Salman, Mattias
  • due: 25 March
  • ngstat
  • arcstat
  • arcinfo

F4: retrieval of job results/output

  • Tester: Marek, Jozef
  • due: 25 March
  • ngget
  • arcget

F5: access to stdout/stderr or server logs

  • Tester: Marek, Jozef
  • due: 25 March
  • ngcat
  • arccat

F6: removal of jobs from CE

  • Tester: Marek, Jozef
  • due: 25 March
  • ngclean
  • arcclean

F6: termination of active jobs

  • Tester: Marek
  • due: 25 March
  • ngkill
  • arckill

F7: proxy renewal for an active job

  • Tester: Uppsala (Samir)
  • due: 25 March
  • ngrenew
  • arcrenew

F8: job failure recovery

  • Tester: Copenhagen
  • due: 1st April
  • ngresume
  • arcresume

F9: job list population via job discovery

  • Tester Uppsala (Salman)
  • due 25 March
  • ngsync
  • arcsync

F10: test utility check

  • Tester: Ivan
  • due: 25 March

The most detailed documentation about ngtest (or arctest) is the man page itself. Even the outdated manuals or the usage (--help) output of the commands contain just a subset of the description of features. Since the role of a functional test is to prove that the software is able to provide the documented features and do it under any circumstances, the base of this test will be the man page and the documented capabilities.

To localize the possible weaknesses it worth to distinguish the features and test them separated from each other. So the following test cases will follow this approach by testing them separately.

ngtest

Documentation errors: [#2303]

  • Referenced man entry doesn't exist: "No entry for arc.conf in section 5 of the manual"
  • -O, -resources stands in the place of -O, -configuration
  • "print this help" stay in the man page which is not THAT help that is referenced

Other errors:

  • When option parsing fails "Invalid options given to arctest" will be printed while executing ngtest (not arctest). [#2304]
Binary usage

Using ngtest [options]: Passed

Using arccli test [options]: Failed Output was: "arccli: command not found" [#2303]

certificates

Command: ngtest -E and ngtest -certificate Expected output: 

Certificate information:

Certificate: ${absolute path to certificate}
Subject name: ${Full DN of certificate separated by slashes}
Valid until: ${Expired or end of validiry as datetime}Thu Oct 20 15:37:46 2011

Proxy: ${absolute path to proxy}
Proxy-subject: ${Full DN of proxy separated by slashes}
Valid for: ${Expired or length of validity as time}

Certificate issuer: ${DN of my certificate's issuer CA}

CA-certificates installed:
${list of DN of installed CA certificates in multiple lines}

Experienced output: 

Certificate information:

Certificate: /home/martoni/.globus/usercert.pem
Subject name: /C=HU/O=NIIF CA/OU=GRID/OU=NIIF/CN=Ivan Marton
Valid until: Thu Oct 20 15:37:46 2011

Proxy: /tmp/x509up_u500
Proxy-subject: /C=HU/O=NIIF CA/OU=GRID/OU=NIIF/CN=Ivan Marton/CN=1659472956
Valid for: Expired

Certificate issuer: /C=HU/O=NIIF/OU=Certificate Authorities/CN=NIIF Root CA

CA-certificates installed:
/C=HU/O=NIIF/OU=Certificate Authorities/CN=NIIF Root CA
/C=IT/O=INFN/CN=INFN CA
/C=HU/O=NIIF/OU=Certificate Authorities/CN=NIIF Root CA
/C=SK/O=SlovakGrid/CN=SlovakGrid CA
/C=IT/O=INFN/CN=INFN CA
/C=SK/O=SlovakGrid/CN=SlovakGrid CA

Result:

  • when certificate, key and proxy are on default locations: Passed

Cosmetic mistake: The list of CA certificates shouldn't be redundant. Now every lines are duplicated. [#2305]

testjob/specify cluster

Command: ngtest -J jobid -c ce1.grid.upjs.sk and ngtest -job jobid -cluster ce1.grid.upjs.sk Expected output: 

Submitting test-job ${Test job id}: 
${Jobdescription belongs to Test job}
Client middleware: ${client specific information}
Job submitted with jobid: ${ID of submitted job}
Cluster: ce1.grid.upjs.sk; middleware: ${server specific information}

Experienced output:

if 1<=jobid<=3 

Submitting test-job 2: 
&("executable" = "/bin/echo" )("arguments" = "hello, grid" )("jobname" = "ARC testjob 2" )("stdout" = "stdout" )("cputime" = "5" )
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
Job submitted with jobid: gsiftp://ce1.grid.upjs.sk:2811/jobs/197481300722006967732817
Cluster: ce1.grid.upjs.sk; middleware: nordugrid-arc-0.8.3.1, globus-unknown

else: 

Illegal testjob-id given

Result:

  • when jobid = -1: Passed
  • when jobid = 0: Failed Output was: "Invalid options given to arctest" [#2306]
  • when jobid = 1: Passed
  • when jobid = 2: Passed
  • when jobid = 3: Passed
  • when jobid = 4: Passed
resources/specify GIIS

Command: ngtest -R -g ldap://arc-emi.grid.upjs.sk:2135/mds-vo-name=ARC-EMI,o=grid and ngtest -resources -giisurl ldap://arc-emi.grid.upjs.sk:2135/mds-vo-name=ARC-EMI,o=grid

Expected output: 

Resource authorizations:

Certificate-subjectname: 
${Full DN of certificate separated by slashes}

Retrieving information...

You are authorized at the following clusters:
${List of DN of authorized clusters or None}

You are authorized at the following storage-elements:
${List of DN of authorized storage elements or None}

Experienced output: 

Resource authorizations:

Certificate-subjectname: 
/C=HU/O=NIIF CA/OU=GRID/OU=NIIF/CN=Ivan Marton

Retrieving information...

You are authorized at the following clusters:
ce1.grid.upjs.sk  pgs03.grid.upjs.sk  rc1.grid.upjs.sk  

You are authorized at the following storage-elements:
None

Result: Passed

configuration

Command: ngtest -O and ngtest -configuration

Expected output: 

Configuration:

ARC_LOCATION: ${location of ARC installation}
GLOBUS_LOCATION: ${location of globus installation}

Top-level GIIS's used:
${list of URL of used GIISs in multiple lines} 

User certificate: ${Full DN of certificate separated by slashes}
Valid until: ${Expired or end of validiry as datetime}Thu Oct 20 15:37:46 2011

Proxy: ${Full DN of proxy separated by slashes}
Valid for: ${Expired or length of validity as time}

Experienced output: 

Configuration:

ARC_LOCATION: /usr
GLOBUS_LOCATION: /usr

Top-level GIIS's used:
ldap://index1.nordugrid.org:2135/O=Grid/Mds-Vo-Name=NorduGrid
ldap://index2.nordugrid.org:2135/O=Grid/Mds-Vo-Name=NorduGrid
ldap://index3.nordugrid.org:2135/O=Grid/Mds-Vo-Name=NorduGrid
ldap://index4.nordugrid.org:2135/O=Grid/Mds-Vo-Name=NorduGrid

User certificate: /C=HU/O=NIIF CA/OU=GRID/OU=NIIF/CN=Ivan Marton
Valid until: 2011-10-20 15:37:46

Proxy: /C=HU/O=NIIF CA/OU=GRID/OU=NIIF/CN=Ivan Marton/CN=1544706721
Valid for: 10 hours, 51 minutes, 17 seconds

Result: Passed

  • Globus location can be reset by GLOBUS_LOCATION environment variable
  • List of GIISes can be set trough ~/.arc/client.conf with values of giis keys in [client] section
vo-membership information

Command: ngtest -V and ngtest -vo

Expected output: 

???

Experienced output: 

ngtest: version 1.0.0b2

Result: Failed [#2307]

version information

Command: ngtest -v and ngtest -version

Expected output: 

ngtest: version ${client version}

Experienced output: 

ngtest: version 1.0.0b2

Result: Passed

help and usage information

Command: ngtest -h and ngtest -help

Expected output: 

${help message and usage information}

Experienced output: 

The expected, that is too long to be pasted here!

Result: Passed

anonymous and gsi-gssapi bind for MDS queries

Command:

  • ngtest -x and ngtest -anonymous
  • ngtest -X and ngtest -gsi

Result: Problems [#2308]

  • anonymous bind cannot be tested alone since this is the default (meaningless option)
  • if both of these options are present (that are mutually exclusives) there are no error messages but the last one will be effective
debug level

Command: ngtest -J 2 -c ce1.grid.upjs.sk -d debuglevel and ngtest -J 2 -c ce1.grid.upjs.sk -debug debuglevel

Expected output: 

${Job submission related messages or error messages on different
verbosity level.}

Experienced output: 

The expected, that is too long to be pasted here!

Result: [#2309]

  • when debuglevel = -4: Failed The output is reticent output of the command not an expection "out of range" type error message
  • when debuglevel = -3: Passed
  • when debuglevel = -2: Passed
  • when debuglevel = -1: Passed
  • when debuglevel = 0: Passed
  • when debuglevel = 1: Passed
  • when debuglevel = 2: Passed
  • when debuglevel = 3: Passed
  • when debuglevel = 4: Failed The output is verbose output of the command not an expection "out of range" type error message
  • when debuglevel = f: Failed The output is: "Can not convert string to int: f" should be more helpful
timeout

Command: ngtest -job 2 -c ce1.grid.upjs.sk -t time and ngtest -job 2 -c ce1.grid.upjs.sk -timeout time

Expected output: 

${timeout message during the job submission}

Experienced output: 

Job submission failed due to: The specified cluster (ce1.grid.upjs.sk) did not return any information.

Result:

  • when 0 < time < necessary for job submission: Passed
  • when time > necessary for job submission: Passed
  • when -t time = -1: Failed it drops a Segmentation fault [#2310]
  • when time = -10, -1.0, f: Passed
dumpxrsl

Command: ngtest -job 2 -c ce1.grid.upjs.sk -dumpxrsl

Expected output: 

Submitting test-job ${test job id} 
${test job description}
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
${modified job description to send to the cluster}

Experienced output: 

Submitting test-job 2: 
&("executable" = "/bin/echo" )("arguments" = "hello, grid" )("jobname" = "ARC testjob 2" )("stdout" = "stdout" )("cputime" = "5" )
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
&("cputime" = "300" )("outputfiles" = ("stdout" "" ) )("clientsoftware" = "nordugrid-arc-1.0.0b2" )("hostname" = "sl5-test1.grid.niif.hu" )("clientxrsl" = "&(""executable"" = ""/bin/echo"" )(""arguments"" = ""hello, grid"" )(""jobname"" = ""ARC testjob 2"" )(""stdout"" = ""stdout"" )(""cputime"" = ""5"" )" )("savestate" = "yes" )("action" = "request" )("queue" = "gridlong" )("stdout" = "stdout" )("jobname" = "ARC testjob 2" )("arguments" = "hello, grid" )("executable" = "/bin/echo" )

Result: Passed

Functional mistake: If cluster is not provided then GIIS will be unnecessarily queried because there won't be any real submission. [#2311]

dryrun

Command: ngtest -job 2 -c ce1.grid.upjs.sk -dumpxrsl -dryrun

Expected output: 

Submitting test-job ${test job id} 
${test job description}
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
${modified job description to send to the cluster containing the formerly missing ("dryrun" = "yes" ) attribute}

Experienced output: 

Submitting test-job 2: 
&("executable" = "/bin/echo" )("arguments" = "hello, grid" )("jobname" = "ARC testjob 2" )("stdout" = "stdout" )("cputime" = "5" )
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
&("cputime" = "300" )("outputfiles" = ("stdout" "" ) )("clientsoftware" = "nordugrid-arc-1.0.0b2" )("hostname" = "sl5-test1.grid.niif.hu" )("clientxrsl" = "&(""executable"" = ""/bin/echo"" )(""arguments"" = ""hello, grid"" )(""jobname"" = ""ARC testjob 2"" )(""stdout"" = ""stdout"" )(""cputime"" = ""5"" )" )("savestate" = "yes" )("action" = "request" )("queue" = "gridlong" )("stdout" = "stdout" )("jobname" = "ARC testjob 2" )("arguments" = "hello, grid" )("executable" = "/bin/echo" )
[martoni@sl5-test1 tmp]$ ngtest -job 2 -c ce1.grid.upjs.sk -dumpxrsl -dryrun
Submitting test-job 2: 
&("executable" = "/bin/echo" )("arguments" = "hello, grid" )("jobname" = "ARC testjob 2" )("stdout" = "stdout" )("cputime" = "5" )
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
&("cputime" = "300" )("dryrun" = "yes" )("outputfiles" = ("stdout" "" ) )("clientsoftware" = "nordugrid-arc-1.0.0b2" )("hostname" = "sl5-test1.grid.niif.hu" )("clientxrsl" = "&(""executable"" = ""/bin/echo"" )(""arguments"" = ""hello, grid"" )(""jobname"" = ""ARC testjob 2"" )(""stdout"" = ""stdout"" )(""cputime"" = ""5"" )" )("savestate" = "yes" )("action" = "request" )("queue" = "gridlong" )("stdout" = "stdout" )("jobname" = "ARC testjob 2" )("arguments" = "hello, grid" )("executable" = "/bin/echo" )

Result: Passed

GIIS list from file

Command: ngtest -giislist giis.list -J 2 and ngtest -G giis.list -J 2

(Where giis.list contains only one line: ldap://arc-emi.grid.upjs.sk:2135/mds-vo-name=ARC-EMI,o=grid and some empty lines.)

Expected output: 

Submitting test-job ${test job id} 
${test job description}
Job submitted with jobid: ${job id}
Cluster: ${cluster from the specified giis}; middleware: ${server side parameters}

Experienced output: 

Submitting test-job 2: 
&("executable" = "/bin/echo" )("arguments" = "hello, grid" )("jobname" = "ARC testjob 2" )("stdout" = "stdout" )("cputime" = "5" )
Client middleware: nordugrid-arc-1.0.0b2, globus-unknown
Job submitted with jobid: gsiftp://ce1.grid.upjs.sk:2811/jobs/104041300736694719890936
Cluster: ce1.grid.upjs.sk; middleware: nordugrid-arc-0.8.3.1, globus-unknown

Result: Passed

Functional mistake: It provides wrong information when using with option -configuration together. [#2312]

Cluster list from file

Command: ngtest -G giis.list -clustlist -cluster.list -J 2 and ngtest -G giis.list -clustlist cluster.list -J 2

(Where giis.list contains only one line: ldap://arc-emi.grid.upjs.sk:2135/mds-vo-name=ARC-EMI,o=grid and some empty lines and cluster.list contains only the name of ce1.grid.upjs.sk)

Result:

  • Select (one of) the cluster(s) specified in the file: Passed
  • Exclude (all of) the cluster(s) specified in the file: Passed
Using custom joblist file

Command: ngtest -job 2 -c ce1.grid.upjs.sk -o filename and ngtest -job 2 -c ce1.grid.upjs.sk -joblist filename

Expected output: The expected output is the same as in case of job submission but with creation of a file specified in filename and with the new job id as the only content.

Result: Passed

runtime

Command: ngtest -runtime time -c ce1.grid.upjs.sk -J 1 -dumpxrsl and ngtest -r time -c ce1.grid.upjs.sk -J 1 -dumpxrsl

  • When using other test job than 1 has no influence: Passed
  • Added to the proper attributes when submitting first test job:
(""CPUTime"" = ""13"" ) and ("arguments" = "11" ) when time=10: Passed
  • Rejecting values where time <=0:
(""CPUTime"" = ""4294967289"" ) and (""arguments"" = ""4294967287"" ) when time=-10: Failed [#2313]

arctest

F10: job description support

  • Tester: Gabor
  • due: 1 April
  • test XRSL
  • test JSDL
  • test JDL

F10: brokering

  • Tester: Uppsala (Salman)
  • due: 1 April
  • test resource discovery, use a JDL that matches "everything"
  • test some of the brokering algorithms

F10: data clis: access to EMI storage elements

  • Tester: Jon (Oslo)
  • due: 1 April
  • arcls
  • arccp
  • arcrm
  • ngls
  • ngcp
  • ngrm

F11: proxy manipulation

  • Tester: Samir will take over in the future
  • due: 25 March
  • ordinary RFC proxy
  • RFC proxy with VOMS extensions

command: arcproxy correct output: 

Your identity: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak 
Enter pass phrase for /home/cernak/.globus/userkey.pem: 
....++++++ 
.......++++++ 
Proxy generation succeeded 
Your proxy is valid until: 2011-03-03 00:10:52

The result of test: 

Your identity: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak 
Enter pass phrase for /home/cernak/.globus/userkey.pem: 
....++++++ 
.......++++++ 
Proxy generation succeeded 
Your proxy is valid until: 2011-03-03 00:10:52

PASSED

command arcproxy -d (FATAL, ERROR, WARNING, INFO, VERBOSE, DEBUG)

arcproxy -d VERBOSE 

INFO: Loading configuration (/etc/arc/client.conf) 
INFO: Configuration (/etc/arc/client.conf) loaded 
INFO: Loading configuration (/home/cernak/.arc/client.conf) 
INFO: Configuration (/home/cernak/.arc/client.conf) loaded 
Your identity: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak 
Enter pass phrase for /home/cernak/.globus/userkey.pem: 
VERBOSE: Certificate verification succeeded 
.++++++ 
.............++++++ 
INFO: Succeeded to sign the proxy certificate 
INFO: Succeeded to verify the signed certificate 
INFO: Output the proxy certificate 
VERBOSE: Certificate verification succeeded 
Proxy generation succeeded 
Your proxy is valid until: 2011-03-03 00:19:41

PASSED

command arcproxy -d INFO 

INFO: Loading configuration (/etc/arc/client.conf) 
INFO: Configuration (/etc/arc/client.conf) loaded 
INFO: Loading configuration (/home/cernak/.arc/client.conf) 
INFO: Configuration (/home/cernak/.arc/client.conf) loaded 
Your identity: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak 
Enter pass phrase for /home/cernak/.globus/userkey.pem: 
...............++++++ 
..++++++ 
INFO: Succeeded to sign the proxy certificate 
INFO: Succeeded to verify the signed certificate 
INFO: Output the proxy certificate 
Proxy generation succeeded 
Your proxy is valid until: 2011-03-03 00:21:00

PASSED

command: arcproxy -v 

arcproxy version 1.0.0b1

PASSED

command arcproxy -I 

Subject: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak/CN=448705739 
Identity: /C=SK/O=SlovakGrid/O=UPJS/CN=Jozef Cernak 
Time left for proxy: 11 hours 45 minutes 48 seconds 
Proxy path: /tmp/x509up_u500 
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy

PASSED

Performance tests

P1: load test of compute cli

Tester: Marek,Jozef due: 25 March submitting 1000 jobs in one client instance

P2: reliability test of compute cli

Tester: Martin Skou due: 25 March

having a client running with unstable environment: dying index services, computing elements stop/start

P1: load test of data cli

Tester: Jon due: 25 March

  • upload/download 5GB file with arccp
    • using file://
    • using https://
    • using gsiftp://
  • upload/download 1000 files with arccp
    • using file://
    • using https://
    • using srm://
    • using gsiftp://

P2: reliability test of data cli

Tester: Jon due: 1 April

tbd

Scalability tests

S1: jobs submission ratio

  • Tester: Jozef
  • due: 1 April
  • test number of jobs submitted sequentially per minute. turn off brokering (direct submission). use some very simple xrsl. no input data staging
  • turn on data staging, turn on brokering, so on...
  • use batch job processing feature (one big xrsl/jsdl containing all the jobs)
  • specifying middleware plugin vs. not-specifying mw plugins


S2: data transfer ratio

  • Tester: Jon
  • due: 1 April

moving lots of small files, or big files

Standard compliance/conformance tests

STD1: JSDL for compute client

  • Tester: Gabor
  • due: May 2011
  • Note that JSDL compliance is also tested in the ARCJSDLParserTest unit test located in source tree at arc1/trunk/src/hed/acc/JobDescriptionParser/ARCJSDLParserTest.cpp.

S2: data standards

  • Tester: Jon
  • due: May 2011

SRM, all the DMCs but RLS...

Inter-component tests

  • Submission to gLite Cream
  • Generation of VOMS proxy certificates
  • interop with EMI data services
Retrieved from "https://wiki.nbi.ku.dk/w/index.php?title=Testing/ARC-Clients_emi1_rc1&oldid=3694"