This wiki is obsolete, see the NorduGrid web pages for up to date information.

ARC v11.05 planning

From NorduGrid
Jump to navigationJump to search

This was the workarea for planning ARC release 11.05.

Release info

  • It'll be a major ARC release
    • with lots of new features
    • major code & packaging restructuring
    • component phase out
  • The (foreseen) name of the ARC Release: 11.03 (code name to be decided)
    • suppose the release date is March 2011.
    • see below for an explanation on the naming/versioning scheme
  • will serve as the ARC contribution for the EMI-1 release

Timing

  • first release preparation tag over svn areas (1.0.0b1): 14.02.11
  • betas available in testing area: achieved, we are already at b3 as of 16.3.2011
  • target release date: April 2011


Tasks

This section lists all the work that needs to be carried out for the release.

Documentation

  • sub-tasks:
    • Release note
    • Migration instructions
    • documentation cleanup
    • webpage (at least the middleware) sync.

Code base

  • sub-tasks:
    • Merge arc0 into arc1 svn tree: done
    • arc1 tree cleanup: done
    • modularization: done

Naming and Versioning

  • subtasks:
    • define consistent proper naming and versioning scheme for ARC releases (resolve the mess around 0.8.x and nox of the past): DONE, see below.

An ARC release is called: ARC Release <release_version) (release_codename). See below for an explanation on the content.

Release versioning

The release_version for a major ARC release is constructed from the date of the release in the following way:

("%02d.%02d",year%100,month) where year and month are the year and month of the release. Minor releases are made from previous major releases by adding a serial integer starting from 1.

Examples of release versions:

  • 12.09 Major release from September 2012
  • 13.01 Major release from January 2013
  • 12.09.1 Minor release based on the 12.09 major release

Note that there is no direct relation between the minor release version and the date that the minor release was released. For instance the 12.09.1 release could be released in May 2013.

The scheme is similar to the Ubuntu LTS releases (see:https://wiki.ubuntu.com/LTS).

Release Codenames

A codename of an ARC release is optional. There are at this moment no rule about the release codenames and it is left for the release manager(s) to choose it.

Packaging

  • subtasks
    • revise the source tarballs
    • revise the source packages
    • revise the binary subpackages
    • revise package content, what goes into which binary package
    • Replace ca-utils with fetch-crl: done
    • Introduce sonames on shared libraries
    • Move the default from /opt/nordugrid -> /usr
    • move (revoke) stuff into/from linux distros
    • define supported platforms
    • setup repositories

Source tarballs

  • source tarballs: name, package version
Name Version Filename Development Origin
nordugrid-arc 1.0.0 nordugrid-arc-1.0.0.tar.gz arc1/branches/1.0.0 NorduGrid SVN arc1/trunk
nordugrid-arc-compat 1.0.0 nordugrid-arc-compat-1.0.0.tar.gz none NorduGrid SVN arc0/branches/compat
nordugrid-arc-doc 1.0.0 nordugrid-arc-doc-1.0.0.tar.gz NorduGrid SVN doc/trunk
arcjobtool 0.4.0 Sourceforge

Source Packages

  • source packages (tarball plus distro specific build instructions)
    • For Fedora 14:
      • nordugrid-arc-1.0.0-1.fc14.src.rpm
      • mingw32-nordugrid-arc-1.0.0-1.fc14.src.rpm
      • nordugrid-arc-compat-1.0.0-1.fc14.src.rpm
      • nordugrid-arc-doc-1.0.0-1.fc14.src.rpm
      • jobtool thingy
    • For Debian:
      • debian src (ver 1.0): .orig.tar.gz, .diff.gz, .dsc
      • debian src (ver 3.0): .orig.tar.gz, .debian.tar.gz, .dsc
      • nordugrid-arc_1.0.0.orig.tar.gz

Binary Packages

  • redhat-like system
    • complete list of binary packages (prefix nordugrid omitted):
      • arc,arc-client,arc-hed,arc-chelonia,arc-doxygen,arc-gridmap-utils,arc-ca-utils,arc-hopi,arc-isis,arc-gridftpd,arc-cache-service,arc-aris,arc-egiis,arc-ldap-monitor,arc-ws-monitor,arc-arex,arc-plugins-needed,arc-plugins-globus,arc-devel,arc-python,arc-java,arc-janitor,arc-doc,arc-compat,arcjobtool
    • Fedora 14 an example full package name: nordugrid-arc-client-1.0.0-1.fc14.x86_64.rpm
  • debian-based system (slightly differs from redhat packages)
    • complete list of binary packages (with shortened names):
  • windows: one installer
    • nordugrid-arc-???-1.0.0-1.exe
  • osx: one installer
Work table
  • The original worktable that was used to decide upon the packages:
    • linux sub packages generated from all the four source packages.
    • The table is redhat-based, debian packages are slightly different
Current Future
client Keep
hed Keep
charon Disable
chelonia Keep
doxygen New
gridmap-utils New
ca-utils New
hopi Keep
isis Keep
gridftpd Keep
cache-service Keep
ldap-infosys Rename: aris
giis Rename: egiis
ldap-monitor Keep
ws-monitor Keep
compiler Remove
delegation Remove
paul Remove
saml2sp Remove
slcs Disable
arex Keep
plugins-needed Keep
plugins-globus Keep
devel Keep
python Keep
java Keep
janitor Keep
doc genrated from nordugrid-arc-doc
compat genrated from nordugrid-arc-compat
jobtool generated from jobtool

Configuration

  • Fix the ARC1 config so that HED services can be configured by humans
    • Aleksandr: the security handler configuration blocks
    • David: datatransfer (A-REX) and loadlimits (A-REX) configuration blocks.
    • Aleksandr and David: A-REX configuration parts not mentioned above.
    • Martin: documentation and templates
  • Make sure arc.conf works with A-REX
    • see A-REX tasks

new client

  • transition to arc* commands, missing features???
  • libarcclient: Java

LDAP infosys

Both ARIS and EGIIS

  • site bdii integration
  • glue2 ldap publishing
  • scalability improvements
  • ....

A-REX

  • phase out GM
  • direct registration to EGIIS
  • transparent transition (configuration tests)
  • co-deployment scenarios (easy config and guide!)
  • scalability, stress-tests


Various other tasks

Validation of all user input

  • xrsl input should be validated
  • arc.conf?

Security overview round 2

  • Fuzzing (throwing random garbage at arc and see if it breaks) (1 week)
  • Infosys should not use root but OpenLDAP user. (Done)
  • SELinux/AppArmor profiles (Long standing issue) (2 weeks)

URL Options Overhaul

Performance improvement

Performance improvement for Index querying (async querying)

  • Update monitor so it doesn't freeze just because one of the services that registers has high latency
  • Update client tools


Important bugs / Features

  • this block needs to be update
  • arcsub misinterprets (queue != "something") in xrsl (2 weeks?) (bug 1915) [done]
  • Improve arcjobtool
    • manage expired certs
    • groups of jobs
    • documentation, how to use
    • fix gui-issues
  • New LRMSes (will not be done)
    • Xtremwebch
    • Easy

Logging

  • Logging should be concentrated in /var/log/*
    • Not in sessiondir/job..errors
    • Not in /var/run/bdiisomething
  • Logs should be properly handled by logrotate (grid-infosys) [done]

Parallel jobs

See the dedicated Parallel_jobs page


Testing and Certification

lots of info to come here. some previous plans (need to be revised)

  • ngtest rewrite (4 weeks) [won't fix]
  • unit tests where applicable, code test coverage for ARC1 > 95% (4 weeks)
    • Documentation for developers on wiki : Martin Skou
  • ubuntu debug packages (1/2 week)
  • testbed improvement (virtual machines) (continous)
    • Red hat support
    • Proper handling of pbs/puppet at boot
    • Support for LSF, Condor, SGE
    • Setup a monitor on master node that can be viewed from outside
    • Add nagios
    • Setup dCache
    • Open up more ports to the outside: web, infosys
    • Continous operation with jobs:
      • Submit jobs
      • check jobs output
      • mixed jobs (randomness, extensible)
  • Nightly builds (http://vls.grid.upjs.sk/testing/index.html)
    • Build
    • Test (like for ARC1)
Retrieved from "https://wiki.nbi.ku.dk/w/index.php?title=ARC_v11.05_planning&oldid=6281"