Testing for update EMI 1.0.1

Organization of testing

Availability of testers during 4.7-15.7.2011

http://doodle.com/4de6fic66hz7rrr7

List of testers

ARC CORE

Marek, Anders, Zsombor

ARC CE

Marek, Anders, Samir, Adrian, Florido

ARC CLIENT

Marek, Anders, Jozef, Salman, Mattias, Jon, Martin Skou, Eva

ARC INFO

Marek, Anders, Adrian, Mattias, Florido

ARC GridFtp

Anders, Marek, Mattias, Salman, Jon

Relevant documents

How to report the results

Tester writes test report. The test report has unique structure

-Description of problem (for example description of bug with link to bugzilla)
-Test scenario (all details needed to reproduce test)
-Infrastructure used to perform test (for example testbed with specification of software)
-Expected results
-Output of test (for example stdout, stderr)
-Comments and proposals (should be used as feedback for developers) 
-The final decision PASSED/FAILED

Test report must be copied in the Test report for EMI comp[onents (wiki) Follow structure of reports recommendation is to keep order of items (for example regression tests bug 1, 2 , 7)

Notes

Regression testing:

If bug is successfully verified change status in bugzilla  to VERIFIED, then provide information about the result in the initial table in the wiki

Priority tasks -actual status

Deploiment tests

Testing deployment of ARC Information System components is priority

The ARC packages (including rc1 of ARC 1.0.1 release) intended for the upcoming EMI-1 update release are now available from the official testing repo

http://emisoft.web.cern.ch/emisoft/dist/EMI/testing/1/sl5/

the corresponding repo file is this

https://twiki.cern.ch/twiki/pub/EMI/EMI1Updates/testing.repo

The ETICS configurations used for producing the packages are under emi-arc_R_1_0_1_0 subsystem configuration in emi/arc ETICS subsystem.

The specific tags used in configuration are:

arc1/tags/1.0.1rc1
arc0/tags/compat_1.0.0
doc/tags/1.0.1
packaging/fedora/nordugrid-arc-meta/tags/1.0.0

Deploiment tests - Marek

EGIIS

Observations:

• 1. the nordugrid-arc-arex package is being installed as part of nordugrid-arc-information-index metapackage. Reported as bug no. 2514

• 2. when installing nordugrid-arc-information-index metapackage the installation of CA packages is not triggered. Should it be? Are the packages mandatory for functional EGIIS service?

• 3. grid-infosys unable to run on non-standard port. When I replaced in the arc.conf the 2135 port with something else, e.g. 5555 and tried to start the grid-infosys daemon it failed with messages:

[root@testbed-emi1 etc]# /etc/init.d/grid-infosys start
Starting grid-infosys:                                     [FAILED]
grid-infosys SLAPD failed to start                         [FAILED]
Error was:                                                 [FAILED]
@(#) $OpenLDAP: slapd 2.3.43 (Mar 10 2011 18:51:22) $
brewbuilder@norob.fnal.gov:/builddir/build/BUILD/openldap-2.3.43/openldap-2.3.43/build-servers/servers/slapd
daemon_init: listen on ldap://*:5555
daemon_init: 1 listeners to open...
ldap_url_parse_ext(ldap://*:5555)
daemon: bind(7) failed errno=13 (Permission denied)
daemon: bind(7) failed errno=13 (Permission denied)
slap_open_listener: failed on ldap://*:5555
slapd stopped.
connections_destroy: nothing to destroy.

I reported that as bug no. 2513.

The config file I used for EGIIS was:

[common]
hostname="testbed-emi1.grid.upjs.sk"
x509_user_key="/etc/grid-security/hostkey.pem"
x509_user_cert="/etc/grid-security/hostcert.pem"
x509_cert_dir="/etc/grid-security/certificates"
[infosys]
overwrite_config="yes"
oldconfsuffix=".oldconfig"
hostname="testbed-emi1.grid.upjs.sk"
port="2135"
debug="1"
slapd_loglevel="1"
slapd_hostnamebind="*"
threads="128"
timelimit="1800"
registrationlog="/var/log/arc/inforegistration.log"
providerlog="/var/log/arc/infoprovider.log"
provider_loglevel="2"
limit_core="0"
limit_nofile=""
user="root"
infosys_compat=enable
infosys_nordugrid=enable
infosys_glue12=disable
infosys_glue2_ldap=disable
bdii_location="/usr"
bdii_tmp_dir="/var/tmp/bdii"
bdii_var_dir="/var/run/bdii"
bdii_log_dir="/var/log/bdii"
bdii_cmd="/etc/init.d/bdii"
bdii_update_cmd="/usr/sbin/bdii-update"
[infosys/index/ARC-EMI-1.0.1]
name="ARC-EMI-1.0.1"
allowreg=* 
[infosys/glue12]
resource_location="Kosice, Slovakia"
resource_latitude="0"
resource_longitude="0"
cpu_scaling_reference_si00=2400
processor_other_description='Cores=3,Benchmark=9.8-HEP-SPEC06'

the EGIIS service installed with RC of 1.0.1 packages is available from testbed-emi1.grid.upjs.sk and can be used for registering any cluster (until we are done with testing). Its grid-monitor is located at http://testbed-emi1.grid.upjs.sk/gridmonitor/loadmon.php

Compute-Element

after Cristina fixed the EMI testing repo, I have tried to install and configure the nordugrid-arc-compute-element. The metapackage did not trigger the installation of CA packages, I think that in this case it should. I reported that as a bug. no. 2519

After successful installation I have tried to register the cluster to the EGIIS running on testbed-emi1 (EMI 1.0.1rc used for testbed-emi1) but I failed. The local ldap tree was always empty. After running the "magic script" I managed to register the cluster. So the problem with infosys still looks to be present in the 1.0.1rc code. The CE successfully filled the local ldap tree with correct info and published it to EGIIS service after BDII version 5.2.4-1.el5 was used.

The configuration used was following:

[common]
pbs_bin_path="/usr/local/bin"
pbs_log_path="/var/spool/torque/server_logs"
hostname="testbed-emi4.grid.upjs.sk"
lrms="pbs"
globus_tcp_port_range="9000,11000"
x509_user_key="/etc/grid-security/hostkey.pem"
x509_user_cert="/etc/grid-security/hostcert.pem"
x509_cert_dir="/etc/grid-security/certificates"
gridmap="/etc/grid-security/grid-mapfile"
[vo]
id="vo_1"
vo="ARC_Community"
source="http://www.nordugrid.org/community.dn"
source="http://vls.grid.upjs.sk/comunity.dn"
source="http://vls.grid.upjs.sk/demo-dn-list.dn"
source="http://www.knowarc.eu/knowarc.dn"
source="vomss://arthur.hep.lu.se:8443/voms/knowarc.eu?/knowarc.eu"
source="vomss://emitestbed07.cnaf.infn.it:8443/voms/testers.eu-emi.eu?/testers.eu-emi.eu"
mapped_unixid="gridtest"
require_issuerdn="no"
[group/users]
name="users"
vo="ARC_Community"
[grid-manager]
controldir="/var/spool/nordugrid/jobstatus"
sessiondir="/scratch/grid"
#sessiondir="/mnt/storage"
scratchdir="/scratch/scratch"
runtimedir="/SOFTWARE/runtime"
cachedir="/scratch/cache"
user="root"
mail="grid.support@somewhere.org"
joblog="/tmp/gm-jobs.log"
securetransfer="no"
debug="5"
[gridftpd]
debug="3"
logfile="/var/log/gridftpd.log"
logsize="100000 2"
pidfile="/var/run/gridftpd.pid"
port="2811"
pluginpath="/usr/lib64/arc"
encryption="no"
allowunknown="no"
maxconnections="200"
#unixmap="nobody:nobody all"
unixmap="gridtest:gridtest all"
[gridftpd/filedir]
plugin="fileplugin.so"
path="/unixacl"
mount="/scratch/tests"
dir="/ nouser read cd dirlist delete create *:* 664:664 mkdir *:* 775:775"
[gridftpd/jobs]
path="/jobs"
plugin="jobplugin.so"
allownew="yes"
[httpsd]
user="root"
debug="2"
logfile="/var/log/httpsd.log"
logsize="100000 2"
pidfile="/var/run/httpsd.pid"
gsiport="8000"
sslport="8001"
#plugin="/opt/nordugrid/lib/se.so"
[se/filedir]
name="filedir"
assigned_storage="filedir"
alias="KE-testbed-emi4-unixacl"
comment="Under test"
type="gridftp"
[infosys]
overwrite_config="yes"
oldconfsuffix=".oldconfig"
hostname="testbed-emi4.grid.upjs.sk"
port="2135"
debug="2"
slapd_loglevel="3"
slapd_hostnamebind="*"
threads="128"
timelimit="1800"
registrationlog="/var/log/inforegistration.log"
providerlog="/var/log/infoprovider.log"
provider_loglevel="3"
limit_core="0"
limit_nofile=""
user="root"
infosys_nordugrid=enable
infosys_glue12=disable
infosys_glue2_ldap=enable
bdii_location="/usr"
bdii_tmp_dir="/var/tmp/bdii"
bdii_var_dir="/var/run/bdii"
bdii_log_dir="/var/log/bdii"
bdii_cmd="/etc/init.d/bdii"
bdii_update_cmd="/usr/sbin/bdii-update"
[cluster]
cluster_alias="testbed-emi4-UPJS"
cluster_location="SK-04001-Kosice"
cluster_owner="UPJS"
clustersupport="marek.kocan@upjs.sk"
lrmsconfig="single job per processor"
homogeneity="True"
architecture="adotf"
nodecpu="AMD64 Processor @ 3500+ MHz"
opsys="SL5 64bit"
nodememory="1024"
middleware="from SVN"
nodeaccess="inbound"
nodeaccess="outbound"
dedicated_node_string="AMD"
gm_mount_point="/jobs"
gm_port="2811"
cachetime="30"
timelimit="30"
sizelimit="10"
[infosys/glue12]
resource_location="Kosice, Slovakia"
resource_latitude="0"
resource_longitude="0"
[infosys/cluster/registration/ARC-EMI-1.0.1]
targethostname="testbed-emi1.grid.upjs.sk"
targetport="2135"
targetsuffix="mds-vo-name=ARC-EMI-1.0.1,o=grid"
regperiod="60"
registranthostname="testbed-emi4.grid.upjs.sk"
registrantport="2135"
[queue/gridlong]
name="gridlong"
homogeneity="True"
scheduling_policy="FIFO"
comment="gridlong"
queue_node_string="AMD"
totalcpus="1"
cachetime="30"
timelimit="30"
sizelimit="5000"
[infosys/se/filedir/registration/sefiledirARC-EMI-1.0.1]
targethostname="testbed-emi1.grid.upjs.sk"
targetport="2135"
targetsuffix="mds-vo-name=ARC-EMI-1.0.1,o=grid"
regperiod="180"

Test	Description	Component	Tester	Result
	clean	CORE	Marek	n/a
	upgrade	CORE	Marek	n/a
	clean	CE	Marek	n/a
	clean	CE	Jozef	FAILED (PASSED with changes in testing.repo)
	upgrade	CE	Marek	n/a
	clean	CLIENT	Jozef	FAILED
	clean	INFO	Florido, Marek	PASSED with BDII 5.2.4-1.el5 - Marek PASSED confirmed - Florido
	upgrade	INFO	Florido, Marek	n/a

Regresion Tests

List of bugs to test (grouped based on priority)/regression testing

BLOCKER

bug	description	component	tester	result
2236	Uploaders hangs	CE	Marek	n/a
2436	YUM repository configuration is wrong	CLIENT	Marek	n/a
2475	XRSLParserTest fails on some 64bit platforms: RHEL 4+5, SL5, Fedora 5+6	CLIENT	Marek	n/a
2482	looping downloader	CLIENT	Marek	n/a

CRITICAL

bug	description	component	tester	result
2301	Uploaders from WS ARC segfault while trying to upload to SRM:// URLs	CE	Jozef	n/a
2472	Period and Time not usable with python for missing methods to get values	CLIENT	Jozef	n/a
2474	renew credentials does not work with arc1 server	CE	Jozef	n/a
2487	Infosystem thrashes the contents of /var/lock/subsys/ when started	INFO	Florido	POSTPONED: the bugfix didn't make it into EMI 1.0.1-0.rc1 . See bug for status

MAJOR

bug	description	component	tester	result
1887	arcstat -a very inefficient	Client	Gabor	PASSED
1979	Configuration issues when running a split GFS / GM setup	Info		n/a
2266	Problem building the nordugrid-arc-1.0.0b1 tarball that was generated in ETICS (swig compatibility)	Client		n/a
2449	ARC 11.05 parsing voms attributes	CE		n/a
2452	ARClib's DownloadDirectory aborts with: globus_list_rest: Assertion `head != 0' failed.	Client		n/a
2463	argget report failure in downloading jobs from ARC0 sites	Client	Gabor	PASSED

normal

bug	description	component	tester	result
2148	"User tools (arccp/rm/ls) should uri encode their arguments before using them"
2219	"Segmentation faults in DataStagingDelivery"
2288	"More informative errors in arcstat when a site is down"	Client	Gabor	on ARC0's CE with 1.0.0 middleware is PASSED
2291	"No pointer to Windows, Mac packages downloads"
2365	"Job submission failed, no more possible targets however server is health"	Client
2370	"multiple arched processes are spawned"
2388	"Logrotate and gridftp and A-REX"
2405	"gridftp dmc is not using anonymous as the default for plain ftp."
2411	"Only SLURM 2.2.1 supported, not newer version."
2418	"Repository for Ubuntu 11.04"
2420	"arcproxy manpage errors"
2425	"Timeout for user uploadable files is too long"
2427	"gm-kick is called by scan-SLURM-job but it is not installed in {$basedir}"
2429	"ldapsearch should escape parentheses"
2430	"ARC shouldn't request a gsiftp URL when it isn't configured to use one"
2431	"ARC should release the TURL after transfer failure"
2434	"No man page - nordugridmap"	CE	Florido	PASSED
2435	"No man page - arc-infoindex-relay & arc-infoindex-server"
2441	"Unable to configure ARC to build with external LCMAPS headers"
2445	"A-REX crashes when using WS-interface"
2453	"arcclean reporting failure at server site"
2455	"arc1 fails to compile with recent libxml2 and xmlsec"
2456	"seg fault from gridftpd when submitting job"
2458	"arcls fails to list files on dcache gridftp server"
2470	"URLs with spaces cannot be cached"
2493	"ngrm command line options descriptions differ"
2494	"jobs get stuck in EXECUTED when resumed after failing in uploading output files"
2496	"Wrong metapackage name in server installation instructions"
2499	"ARC build problems"
2407	"No check for Python.h is made in ./configure, but the python bindings are built"
2412	"Downloading fails from ARC gridftpd 0.8.1 site when using cache."
2454	"Debian and Ubuntu repo examples have testing instead of experimental"
2403	"arc.conf.template refers to arc server but is in package nordugrid-arc-client"
2461	"GGUS-Ticket-ID: #71248 EMI1: arc-ce documentation issues"
1291	"ability to use srm over https through url option"
2015	"merged standalone to contain both new and old client tools"
2076	"Distribution of binary release packages needs improvements"

Performance tests

P1 Submission of 1000 complex jobs

The evaluation of the test: PASSED

P2 Download of 1000 jobs

The evaluation of the test: PASSED

The new bug reported 2521

Tests of reliability

Reliability of obtaining results (Jozef)

Test report EMI components

Reports for the ARC Core components

Reports for the ARC Computing Element

Reports for the ARC Client components

Reports for the ARC Information System components

Reports for the ARC GridFTP server