Testing for update EMI 2.0.0

Deadlines

EMI 2 (Matterhorn) - Activity:

https://twiki.cern.ch/twiki/bin/view/EMI/EMI-2

Delivered in EMI-2

CE and clients supporting EMI-WS

CEs with EMI accounting record -Gabor

Consumer side GLUE2 support

Transparent integration with AAI

ARGUS integration with compute and data services

EMI Service Registry

EMI_authlib

Complete set of Nagios probes

Debian and SL support

ARC gridftp server v. 2.0.0

ARC CE v. 2.0.0

ARC Infosys v. 2.0.0

ARC Clients v. 2.0.0

ARC Core v. 2.0.0

EMI2 RC3 tagged with rev. no. 24075

Test plan

Media: ARC_test_plan_v0.5.doc

Integration testing in EMI:

https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1TaskForceIntegrationTesting

Test report

https://twiki.cern.ch/twiki/bin/view/EMI/EMITestReport

Test recording procedures

In relation to test results recording I would like to ask anyone who will participate in testing to record his/her results using form from http://arc-emi.grid.upjs.sk/testing/testResult.php . For regression tests we are asked to use some unique IDs, so please use the bug number as this unique ID in the test name field e.g. like 'Bug no. 1783 - arcstat asks for passphrase even if proxy exists'. If you have recorded your regression test then do not forget to update the bugzilla record. Since EMI is tracking information if some resolved bug had a regression test, it was agreed that after the test is recorded the attachment with name 'regression_test' including link pointing to test result recorded (link is generated after the test result is saved) shall be added to a particular bugzilla record. For other types of tests (like functionality, performance, scalability, ...) following form from http://arc-emi.grid.upjs.sk/testing/testResult.php shall be enough.

Nagios plugins

https://twiki.cern.ch/twiki/bin/view/EMI/EticsPluginsFramework

Deployment test procedure

Description of deployment test:

Clean Installation

The deployment test for clean installation shall be performed on a machine with freshly installed OS (no extra dependencies required by any of ARC components shall be already installed)

- Configure the yum/apt with Etics repository (https://twiki.cern.ch/twiki/bin/view/EMI/EMI-2#ETICS_project_configurations) as a main source of packages and record the repository configuration details (will be needed for test report)

- Install particular ARC component and record the command used for installation together with the list of packages that were installed by this command.

- Get the out-of-box configuration file (http://svn.nordugrid.org/trac/nordugrid/browser/doc/trunk/examples or http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/tests/echo/echo_service.xml.example.in or install nordugrid-arc-doc package and use the config files from usr/share/doc/nordugrid-arc-doc/examples) available for particular ARC component (for ARC Clients no configuration is needed), edit it if necessary and try to start the ARC service

- Make basic verification of ARC service
  - ARC CE

submit arctest -J 1 -c <clustername>  , arctest -J 2 -c <clustername>, arctest -J 3 -c <clustername>

the ARC CE deployment is declared as successful if all three jobs finished succesfully.

- - ARC Information System
    - EGIIS - register at least one ARC CE (installed using production version of ARC packages) with the EGIIS and issue following command against this EGIIS service:

ldapsearch -h <egiishostname> -p 2135 -x -b 'mds-vo-name=<EGIIS_name>,o=grid' -s base giisregistrationstatus

The test is successful if you get the list of ARC CEs registered with some additional cluster information printed

stop all ARC CEs registered to the EGIIS and issue the ldapsearch command from above

The test is successful if the printed output does not contain any of the ARC CE clusters

The EGIIS deployment is declared as successful if both EGIIS tests are successful

- - - ARIS - against the ARC CE started in step 3 issue the following ldapsearch command:

ldapsearch -h <clustername> -p 2135 -x -b 'mds-vo-name=local,o=grid' The ARIS deployment is declared as successful when you get cluster information printed after the ldapsearch command is run

- - GridFTP - upload a file to GridFTP server started in step 3 using arccp command, list the content of the GridFTP server using arcls command, download file from GridFTP server using arccp command and remove the uploded file using arcrm command. The arc* commands used for this test should be installed using production version of ARC packages

- The GridFTP deployment is declared as successful when all above listed operations finish successfully

- - ARC Core - For ARC Core component the Echo service shall be used with at least two different set ups. With TLS and without TLS layer. The verification shall be done using arcecho command.

- The ARC Core deployment is declared as successful when echo responses are obtained from both Echo services.

- - ARC Clients - To declare the ARC Clients deployment as successful at least following commands shall be run using the default configuration (no changes to client.conf):

arcproxy, arcproxy -I, arcsub -c <arc-emi_cluster>, arcstat -a , arcget -k -a , arcclean -a - The services used for ARC Clients verification may be taken from ARC-EMI testbeds

Upgrade installation

The deployment test for Upgrade Installation shall be done identically to Clean Installation with a difference that the upgrade installation of particular ARC component is done on a machine that already has the ARC component installed using latest production version of ARC packages and the commands for step 2 are upgrade commands and not install commands.

The verification is done the same way as for Clean Installation

Unit test coverage

Code analysis

Unit tests

Will be provided by Anders

Organization of testing

The main difference of current testing is using of a new tool to store test report. Each tester is coauthor of test report. The status of testing is monitored via online interface and relevant part of final test reports are available on one click.

The successful results are not repeated if a new RC of 2.0.0 is issued.

To start testing a RC update from NorduGrid repository is taken.

Code repo

ETICS EMI-2 repos:

http://emisoft.web.cern.ch/emisoft/dist/EMI/2/

At the start no ETICS packages are available. 2.0.0 RC1 is available in Nordugrid repo.

http://download.nordugrid.org/repos/latest/redhat/el5/x86_64/testing/RPMS/
http://download.nordugrid.org/repos/latest/redhat/el6/x86_64/testing/RPMS/

Availability of testers

Doodle page: http://www.doodle.com/dhwbanywhpqhwyge

Communication channels

Communication between testers, testers and developers will take place using various channels.

Wiki - It serves for sharing basic and important information, for example list and availability of testers, important dates and detailed test plan.

Database of test reports - All test reports must be submitted using online form which is compliant with EMI testing policies.

Form to submit test report: http://arc-emi.grid.upjs.sk/testing/testResult.php Interface to query performed tests: http://arc-emi.grid.upjs.sk/testing/searchResults.php

Skype

e-mail

Priorities of testing

Regression testing

List of rfc updated 24.January 2012

features

bug_id bug_severity priority op_sys short_desc tester result

http://bugzilla.nordugrid.org/show_bug.cgi?id=2438 enhancement P3 Linux arc.conf.template should have all non-essential fields commented out Jozef-verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2667 enhancement P3 Linux Use Python's __nonzero__ method to wrap C++'s "operator bool"

http://bugzilla.nordugrid.org/show_bug.cgi?id=2466 enhancement P3 Linux A better cache locking solution when multiple hosts share a cache

http://bugzilla.nordugrid.org/show_bug.cgi?id=2569 enhancement P3 Linux new data-staging queue too small and should be configurable

http://bugzilla.nordugrid.org/show_bug.cgi?id=2603 enhancement P3 Linux Add option to force source check every time cache file is requested Marek

http://bugzilla.nordugrid.org/show_bug.cgi?id=2695 enhancement P3 Linux Use all transfer slots even when one share has a small queue

http://bugzilla.nordugrid.org/show_bug.cgi?id=2708 enhancement P3 Linux Multi-host DTR should be able to direct transfers to specific nodes

http://bugzilla.nordugrid.org/show_bug.cgi?id=2316 enhancement P3 Linux infosys displays truncated information for failed transfers Marek - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2621 enhancement P3 Linux arcproxy must be able to provide the same information as voms-proxy-info -all

http://bugzilla.nordugrid.org/show_bug.cgi?id=2009 enhancement P4 All ngget / arcget option --force needed

http://bugzilla.nordugrid.org/show_bug.cgi?id=2278 feature request P2 Linux Create arcmkdir Marek - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=1928 feature request P3 All Client tools should check proxy validity Marek - verified

Bugs

bug_id bug_severity priority op_sys short_desc Tester Resut

http://bugzilla.nordugrid.org/show_bug.cgi?id=2711 blocker P3 Linux SLURM job submit script does not start if configured as documented Marek - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2697 blocker P3 Linux Ubuntu repos for 11.05 are broken R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2719 blocker P3 Linux slurm memory settings wrong

http://bugzilla.nordugrid.org/show_bug.cgi?id=2660 blocker P3 Windows 7 arcproxy not creating a proxy R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2524 critical P1 Linux provide out-of-box configuration examples for server-side ARC (CE, EGIIS and SE) Marek - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2509 critical P1 All Information system manual/guide is needed

http://bugzilla.nordugrid.org/show_bug.cgi?id=2477 critical P1 Linux Distribute ARC nagios plugins together with the next ARC release

http://bugzilla.nordugrid.org/show_bug.cgi?id=2672 critical P3 Linux A-REX wakeup thread is stuck

http://bugzilla.nordugrid.org/show_bug.cgi?id=2698 major P3 Linux arccp/rm/ls tries to incorrectly access the hostcert when run as root Marek

http://bugzilla.nordugrid.org/show_bug.cgi?id=2032 major P3 Red Hat Enterprise Linux 5 Using BDIIv5 with the information system db log files are not deleted Jozef -verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2657 major P3 Scientific Linux 5 ARC CE does not publish cluster attribute and does not show in grid-monitor Marek - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2646 major P3 Linux nordugridmap does not apply specified filters R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2444 major P3 Linux arcstat -a doesn't ask the status of all jobs if there is a problem for one of the jobs

http://bugzilla.nordugrid.org/show_bug.cgi?id=2671 major P3 Linux LL does not show queued job entries

http://bugzilla.nordugrid.org/show_bug.cgi?id=2750 major P3 Scientific Linux 5 Jura client will fail by a simple usage R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=1827 normal P2 Linux downloader retry strategy is not optimal for cache locks

http://bugzilla.nordugrid.org/show_bug.cgi?id=2421 normal P3 Linux RLS download deadlocks downloader

http://bugzilla.nordugrid.org/show_bug.cgi?id=2558 normal P3 Scientific Linux 5 locking of CE

http://bugzilla.nordugrid.org/show_bug.cgi?id=2677 normal P3 Ubuntu Linux arcls segfaults on dCache gsiftp when not authorised

http://bugzilla.nordugrid.org/show_bug.cgi?id=2680 normal P3 Linux Transfer timeouts when copying large files (>100MB) from http

http://bugzilla.nordugrid.org/show_bug.cgi?id=2681 normal P3 Linux Auth and mapping plugins stderr does not captured to ARC logs

http://bugzilla.nordugrid.org/show_bug.cgi?id=2715 normal P3 Linux arccat segfaults with several jobs if the files of the first jobs are not available

http://bugzilla.nordugrid.org/show_bug.cgi?id=2717 normal P3 Linux SimpleCondition::broadcast() only wakes up one thread

http://bugzilla.nordugrid.org/show_bug.cgi?id=2723 normal P3 Linux a [vo] [groups] configuration makes arched hang when contacted via arcwsrf

http://bugzilla.nordugrid.org/show_bug.cgi?id=2661 normal P3 Linux BDII_BIND is not printed to bdii.conf

http://bugzilla.nordugrid.org/show_bug.cgi?id=2659 normal P3 All Files erased on cluster despite srm problem

http://bugzilla.nordugrid.org/show_bug.cgi?id=2688 normal P3 Linux Authplugins at PREPARING and FINISHING don't work with new data staging

http://bugzilla.nordugrid.org/show_bug.cgi?id=2700 normal P3 Linux arccp cannot handle certain characters in URLs

http://bugzilla.nordugrid.org/show_bug.cgi?id=2712 normal P3 Linux Safe restart of DTR uploads after A-REX crash or incomplete stop

http://bugzilla.nordugrid.org/show_bug.cgi?id=2300 normal P3 Linux Incomplete CE info in ARIS (LDAP)

http://bugzilla.nordugrid.org/show_bug.cgi?id=2373 normal P3 Linux publishing glue12 breaks when [infosys/glue12] contains some special characters

http://bugzilla.nordugrid.org/show_bug.cgi?id=2664 normal P3 Windows 7 arcget does not download any files

http://bugzilla.nordugrid.org/show_bug.cgi?id=1821 normal P3 All ngsub exit with code 0 when submission fails due to cluster not returning any information

http://bugzilla.nordugrid.org/show_bug.cgi?id=2343 normal P3 Red Hat Enterprise Linux 5 JSDL to Inner representaion: JobName

http://bugzilla.nordugrid.org/show_bug.cgi?id=2645 normal P3 All The libarcclient library depends on the job list file

http://bugzilla.nordugrid.org/show_bug.cgi?id=2678 normal P3 Linux arcsub uses the wrong unit for <MemoryLimit> element.

http://bugzilla.nordugrid.org/show_bug.cgi?id=2510 normal P3 Linux GGUS-Ticket-ID: #72260 Nagios probe org.arc.LFC-ops fails with ARC 1.0.0

http://bugzilla.nordugrid.org/show_bug.cgi?id=2669 normal P3 Linux arcproxy does not use default cacertdir folder

http://bugzilla.nordugrid.org/show_bug.cgi?id=2638 normal P3 Linux latex dependencies on arc_ce_sysadm_guide

http://bugzilla.nordugrid.org/show_bug.cgi?id=2733 normal P3 Linux ExitStatus element is missing from the generated CAR R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2734 normal P3 Linux LocalJobId is not present in the generated CAR R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2735 normal P3 Linux Always generate LocalUserId element into the CAR R. Gabor - verified

http://bugzilla.nordugrid.org/show_bug.cgi?id=2518 normal P4 All GGUS-Ticket-ID: #72415 SAM Nagios ARC probes allocates too little memory

DO NOT FORGET TO UPDATE THE BUG REPORT ONCE YOU HAVE VERFIED THE BUG. SEE THIS INFO

Deep testing of CE

Notes

The section are place where detailed information about test plan and notes should be collected. These parts of text are not copied in the final report.

Notes for the ARC Core components

Tesk in EMI: https://savannah.cern.ch/task/?23304

Notes for the ARC Computing Element

Task in EMI: https://savannah.cern.ch/task/?23302

server details

hardware rc0.grid.upjs.sk and rc1.grid.upjs.sk

CPU:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 12
model name      : AMD Athlon(tm) 64 Processor 3000+
stepping        : 0
cpu MHz         : 2000.107
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
bogomips        : 4000.21
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

MEM:

MemTotal:       895036 kB

Software

OS:

LSB Version:	:core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
Distributor ID:	ScientificSL
Description:	Scientific Linux SL release 5.7 (Boron)
Release:	5.7
Codename:	Boron

arc.conf

[common]
x509_user_key="/etc/grid-security/hostkey.pem"
x509_user_cert="/etc/grid-security/hostcert.pem"
x509_cert_dir="/etc/grid-security/certificates" 
gridmap="/etc/grid-security/grid-mapfile"
lrms="fork"
globus_tcp_port_range="9000,11000"

[vo]
id="vo_1"
vo="ARC_Community"
source="http://www.nordugrid.org/community.dn"
source="http://vls.grid.upjs.sk/comunity.dn"  
source="http://vls.grid.upjs.sk/demo-dn-list.dn"
source="http://www.knowarc.eu/knowarc.dn"
source="vomss://arthur.hep.lu.se:8443/voms/knowarc.eu?/knowarc.eu"
source="vomss://emitestbed07.cnaf.infn.it:8443/voms/testers.eu-emi.eu?/testers.eu-emi.
mapped_unixid="gridtest"
require_issuerdn="no"   
    
[grid-manager]
user="root"   
controldir="/tmp/jobstatus"
sessiondir="/tmp/grid"
debug="5"
logfile="/tmp/grid-manager.log"
pidfile="/tmp/grid-manager.pid"
mail="grid.support@somewhere.org"
joblog="/tmp/gm-jobs.log"
arex_mount_point="https://rc1.grid.upjs.sk:50000/arex"
enable_emies_interface="yes"
default_queue="fork fork"

# gridftp server config
[gridftpd]
user="root"
debug="3"  
logfile="/tmp/gridftpd.log"
pidfile="/tmp/gridftpd.pid"
port="2811"
allowunknown="no"

# job submission interface via gridftp
[gridftpd/jobs]
path="/jobs"   
plugin="jobplugin.so"
allownew="yes"
    
# openldap server config
[infosys]
user="root"
overwrite_config="yes"
port="2135"
debug="1"  
slapd_loglevel="0"
registrationlog="/tmp/inforegistration.log"
providerlog="/tmp/infoprovider.log"
provider_loglevel="2"
    
# infosys view of the computing cluster (service)
[cluster]
cluster_alias="MINIMAL Computing Element-rc1"
comment="This is a minimal out-of-box CE setup"
homogeneity="True"
architecture="adotf"
nodeaccess="inbound"
nodeaccess="outbound"
    
# infosys view of the queue behind the computing service,
# every CE needs at least one queue
[queue/fork]
name="fork" 
fork_job_limit="cpunumber"
homogeneity="True"
scheduling_policy="FIFO"
comment="This queue is nothing more than a fork host"
nodecpu="adotf"
architecture="adotf"
    
    
#cluster registration block
[infosys/cluster/registration/clustertoARC-EMI]
targethostname="arc-emi.grid.upjs.sk"
targetport="2135"
targetsuffix="mds-vo-name=ARC-EMI,o=grid"
regperiod="120"

Notes for the ARC Client components

Task in EMI: https://savannah.cern.ch/task/?23303

Notes for the ARC Information System components

Task in EMI https://savannah.cern.ch/task/?23306

ldapsearch -x -H "ldap://rc0.grid.upjs.sk:2135" -b "mds-vo-name=local,o=grid"

Notes for the ARC GridFTP server

Tesk in EMI: https://savannah.cern.ch/task/?23305

arc.cong

[common]
x509_user_key="/etc/grid-security/hostkey.pem"
x509_user_cert="/etc/grid-security/hostcert.pem"
x509_cert_dir="/etc/grid-security/certificates"
gridmap="/etc/grid-security/grid-mapfile"
globus_tcp_port_range="9000,11000"

#gridftp server config
[gridftpd]
user="root"
debug="3"
logfile="/tmp/gridftpd.log"
logsize="100000 2"
pidfile="/tmp/gridftpd.pid"
port="2811"
allowunknown="no"
pluginpath="/usr/lib64/arc/"
    
#config of the "exported" directory
[gridftpd/unixacl]
plugin="fileplugin.so"
path="/unixacl"
mount="/se/unixacl"
dir="/ nouser read cd dirlist delete create *:* 664:664 mkdir *:* 775:775"

Functional tests

Standard compliance/conformance tests

Test results and details

test results and details of tests are stored in DB:

http://arc-emi.grid.upjs.sk/tests.php

Static test results recording http://arc-emi.grid.upjs.sk/testing/testResult.php ( Search static test results http://arc-emi.grid.upjs.sk/testing/searchResults.php )