This wiki is obsolete, see the NorduGrid web pages for up to date information.

Release management

From NorduGrid
Jump to navigationJump to search

This page collects general information concerning the release procedure of ARC software.

Background

  • An ARC release is defined as a set of source packages AND the corresponding binary packages on the supported platforms (see Release packaging)
  • ARC middleware has been used in production since May 2002
  • ARC development is coordinated by the NorduGrid Collaboration
  • ARC is developed by several projects and individual contributors; contributions are from a diverse, active, inhomogeneous community
  • ARC serves the user communities of many scientific domains all over the world.
  • ARC releases have been created with the strong emphasis on multiplatform support, Linux distribution independence and OS independence (Linux, Windows, Mac, Solaris), non-intrusiveness and ease of operation
  • Since May 2011, ARC releases are part of EMI releases and follow EMI policies and rules

Important links

Less important (old) links

Actors

  • system architects: higher level thinkers, key developers with full overview of system components
  • developers: working for different projects at different locations, diverse background, different level of engagement
    • core: full overview of a certain crucial part of the software
    • component developer: responsible for a certain module
    • task developer: shorter term assignment, most of the time student projects
    • contributor: someone providing patches
  • system administrators: end of the deployment chain guys with real knowledge of the software
  • infrastructure decision makers: high-level actors influencing software deployment on infrastructure
  • users:
    • organized as larger communities ("big VO")
    • individual users ("small guys")
  • code keeper: the source repository maintainer, a person responsible for code repository on a high level, consistency of the source code environment
  • release manager: a person responsible for the complete life cycle of a major release including eventual minor sub-releases. appointed by coordination group
  • bug manager: a person responsible for monitoring issues in Bugzilla
  • binarians: group of developers responsible for the binary build, software distribution, packaging, binary repositories, download areas
  • test team: people defining and implementing tests, maintaining test infrastructure, coordinating testing activities, executing the tests
  • Technical Coordination Group: defines middle and longer term strategies, does software coordination and monitoring
    • includes architects, developers, user representatives, decision makers

Relevant people

In no particular order:

Name Expertise, responsibility
NorduGrid
Aleksandr Konstantinov A-REX, HED, grid-manager, security tools
Anders Wäänänen Packaging, autotools, access to SVN/Wiki/Bugzilla
Andrej Filipcic Testing, ATLAS, ARC Control Tower (aCT)
Balazs Konya Overall coordination, link to EGI
David Cameron dCache, data clients, staging, cache, aCT
Maiken Pedersen Release manager, aCT
Marek Kočan ARC testing
Jozef Černák ARC testing
Mattias Ellert packaging and builds, EPEL and Debian uploads, Globus
Martin Skou Andersen compute client
Florido Paganelli infosystem, EMI-ES, GLUE2, documentation
Oxana Smirnova Bugzilla, Web site, documentation
Andrii Salnikov nordugridmap
EGI, NeIC (NDGF)
Mattias Wadenstein Nordic Tier1 coordinator, central services, accounting
Erik Edelmann CMS (arc), job priorities, SGAS
Gerd Behrmann dCache
Jens Larsson Early Adopter, application support, FTS admin
Jonas Lindemann arcjobtool, GUI
Leif Nixon security expert
Ulf Tiggerstedt Early Adopter, application support
Sergio Maffioletti Early Adopter, application support
Petter Urkedal Nagios tests
EMI, URT
Cristina Aiftimiei EMI Release manager
Peter Solagna EGI, UMD Release manager
Non-affiliated/retired
Ali Gholami ARC-ARGUS integration, authorization, accounting, data management
Henrik Thostrup Jensen SGAS, ACIX

Development environment

  • Most code is kept in NorduGrid SVN, spread over several SVN trees. ARC has a non-monolitic source code tree. Some code is kept externally (Sourceforge, github).
  • SVN read access is provided for everyone
  • SVN commit access is regulated and the commit rights are granted by the the coordination group
    • One needs a valid Grid certificate to commit to NorduGrid SVN
    • entry-level access is given to a dedicated area called "workarea". This is the place for initial code uploads.
    • feature branches are created for implementing larger-scale new functionality and commit access is granted to assigned developers
    • trunk is kept for main synchronized development and comes with a more restricted access policy
    • release branches represent the most stable areas and commits to the branches are closely monitored, towards the end of the release procedure even controlled by the release coordinator
  • ARC development is governed by a set of simple rules
  • Bugzilla is used to report software defects and to act upon them
    • feature requests are also collected in Bugzilla
    • bugs, including feature requests are periodically reviewed and prioritized by the core group
  • NorduGrid Wiki is used as a workarea:
    • to collect development plans, technical drafts, ideas
    • release coordination
    • configuration templates
    • quick guides, HOWTOs
  • community forum is nordugrid-discuss mailing list
    • high traffic technical list
  • Ticketing system
    • GGUS is used to provide user support
    • eventual software defects discovered during the user assistance process are recorded in Bugzilla by the support staff

Nightly builds

There are two different nightly build systems:

  • A nightly "standard" ARC build (in Kosice) which is in fact running the revision tests thus usually the last revision from a given day corresponds to the nightly build
  • A nightly "package" NorduGrid build (in Copenhagen).

The standard ARC build does each build step separately while the package build does everything from tarball/src.rpm - including creating pristine environments for each package build. This system is identical to the release builds. So in summary the nightly "package" build really checks the whole build and package chain for release packages

Release categories

See also Release versioning and Release packaging

  • A NorduGrid ARC release consists of a set of source packages
  • A certain source package is identified by its version number, <major>.<minor>.<bugfix>
    • E.g., NorduGrid ARC 4.1.0 or NorduGrid ARC Documents 1.3.4
  • major release
    • longer term planning
    • 3-6 months preparation
    • alpha, beta, rcx test releases
    • introduces new components, features
    • obsoletes components
    • may break backward compatibility
    • scheduled release
    • this release bumps the major number in the version number of the package causing the major release
  • minor release
    • planned bugfix release
    • max one month preparation
    • no new components
    • only minor new features and/or significant bugfixes
    • scheduled release
    • this release bumps the minor number in the version number of the package causing the minor release
  • bugfix release
    • planned bugfix release
    • max one month preparation
    • no new features, no new components
    • only a limited number of less significant fixes
    • scheduled release
    • this release bumps the bugfix number in the version number of the package causing the bugfix release
  • emergency release
    • unplanned urgent release to fix a security issue or a critical bug
    • maximum two weeks preparation
  • binary update of a release
    • sometimes due to a change in the external dependencies ARC releases have to be rebuilt.
    • This "semi release" does not affect the ARC source code, no re-taging takes place, only new binaries are built
    • can happen for both major, minor or emergency releases

Release Manager tasks

The following tasks constitute daily activities of the Release Manager.

Deciding on new features

Release Manager asks the NorduGrid Technical Coordination Group to decide whether to include new features, and which. Feature requests normally come through Bugzilla, but other communication channels are also possible.

Deciding on commits

Developers know best which commits are important, but they sometimes forget commits and/or commits have side effects, so they need to be test-built and tested quite often. Close discussion with relevant developers is very important. There exist a Skype chat-room where most of the developers are present.

Testing

Testing needs to be done on several Linux distributions. This is very important since ARC depends on a a lot of software that has different versions on different distributions. Also, they don’t have the same file structure. EL/CentOS/Scientific Linux (from now called RHEL) version 6.x architecture x86_64 is The Most Important version. If you need to make a decision on whether a feature/commit should go in or if you don’t have time to do proper testing or need to release software that breaks on platforms. This needs to be remembered. The order after that is probably: RHEL 5, Latest Ubuntu LTS, Latest Ubuntu. Architecture does not matter. After this comes the group: MacOS, MS Windows, Fedora other Ubuntu. After this comes the rest. RHEL6 comes from UMD, the rest is number of users. Some OSes are more used server-side and some client-side.

Tagging

Release candidate (numbered), Final version tags.

In general, 2-3 Release candidates should precede a Final version. This is because: First version will mainly involve testing of packaging (few testers), then problems are found in second test-release which need at least another test-release to verify that it is solved. Beware of the latest test-release, it should not bee too close to Final since problems will almost always be found in it.

Releasing (checklist)

  • Release pages in Wiki
  • Prepare lists of fixed bugs
  • Prepare lists of new features(if relevant)
  • Prepare lists of backward-incompatible changes (if relevant)
  • Decide which tags will constitute the release and list them:
    • ARC
    • Documents
    • Metapackages
    • Nagios probes
    • CAnL-C++
  • Ensure that there is a source package (bundle)
  • Update package VERSION in SVN (arc1)
    • Update trunk/src/hed/libs/common/ArcVersion.h and merge to corresponding branch (release manager)
    • Update branches/X.Y/VERSION, branches/X.Y/debian/changelog.deb.in, branches/X.Y/nordugrid-arc.spec.in (package manager)
  • Tag:
    • ARC code in SVN (arc1)
    • Documents in SVN (if relevant)
    • Nagios probes (if relevant)
    • Metapackages (if relevant)
    • CAnL-C++ (if relevant)
  • Check tags in external repositories:
    • arcgui
  • Ensure there are consistent binary packages in agreed repositories and linked from the release page
  • Ensure there are installation instructions
  • Ensure there is release documentation (preferably as one HTML page), e.g. release notes
  • Ensure there is corresponding release version in Bugzilla
  • Make sure the release is tested: define test criteria (e.g. tested at 3 different sites around the world) and provides a test report
  • Notify nordugrid-core
  • Announce release on the mailing lists (see below)
    • Send out a warning to nordugrid-discuss and nordugrid-site-managers mailing lists about turning off auto-upgrade (if necessary)
  • Security review
  • Status check

See also detailed release procedures below.

Release announcement

A message describing the new release (possibly taken from release notes and/or referring to release notes) is circulated to nordugrid-discuss and nordugrid-site-managers mailing lists. The information and release notes are then published via the following channels:

The following Web pages are updated:

Release procedures

Scheduled release procedure

  • Major/minor/bugfix releases follow a scheduled release procedure
    • The contents of major ARC releases are trying to follow a high level ARC development roadmap
    • Minor releases contain only significant bugfixes and minor features
    • Bugfix releases contain only minor bugfixes
  • Planning phase (community process)
    • overview of external dependencies
    • Prioritizing bugs to be fixed
    • collecting requests for new features (only for major release)
    • consider adding newly developed components to the code base (only for major release)
    • add support for new platforms (only for major release)
    • introduce changes in the packaging, software distribution (only for major release)
  • Decision phase (carried out by the Technical Coordination Group)
    • decision takes into account the status (maturity) of release candidate components
    • the release content is defined in terms of components and bugs to be fixed
    • supported platforms decided
    • time planning of the release preparation
    • appointing release coordinator testing team, binarian
    • informing urt-discuss@mailman.egi.eu, and hoping someone creates release tasks in UMD Savannah
  • Preparation of the release (release coordinator, developer and testing team)
    • Creation of the release branch (only for major release)
    • Documentation updates, Release notes
    • Release candidate(s) are created from the branch
  • Testing and certification
    • Release candidate is deployed on the test infrastructure
    • The testing team executes all the test suits
    • If possible, larger testing campaigns are organized using the release candidate (involving users and sysadmins outside the testing team)
    • Test reports and certification reports are submitted to UMD together with release announcement
  • Release is ready if
    • All the critical bugs recorded in the Bugzilla are closed
    • Functionality decided in planning phase is successfully tested on the release candidate
    • Binary packages are available for all supported platforms
    • Documentation reflecting changes from the previous release is in place
    • Relevant Bugzilla tags are in place

Emergency release procedure

  • running into a MAJOR ISSUE: a software defect considered critical or a security whole was discovered during
    • everyday usage scenario
    • bug fixing, debuging
    • regular testing
    • triggered by external dependency
  • the reporting of the MAJOR ISSUE can happen through several channels: Bugzilla, mailing list, direct contact to Technical Coordination Group members
  • The decision whether ISSUE requires a emergency release is taken by the NorduGrid Technical Coordination Group. Options:
    • prepare an emergency release
    • prepare only a patch offering partial solution only for some of the platforms
    • don't prepare an emergency release but fix the issue with the next scheduled (minor/major/bugfix) release
  • The emergency release is being prepared:
    • release coordinator appointed by Technical Coordination Group
    • identify target branch and create a emergency release branch out of that, decide upon the release number
    • investigate the problem, create a strategy to resolve the issue
    • once the fix is available the Technical Coordination Group appoints an independent tester
    • after successful testing the release coordinator completes the process by
      • updating release notes reflecting changes (fix or workaround)
      • creating tag (deadline should be announced to developers), packages builds
      • informing UMD Release Coordinator
      • populating repositories
  • Emergency release is ready
    • the entire release procedure is closely controlled by the release coordinator
    • the process can't take more time than two weeks

Policies

  • ARC packages pushed to distribution repositories must always be certified and taken from EMI releases

placeholder

  • these are not yet properly covered:
    • testing,
    • external software,
    • integration,
    • rollout tests,
    • inclusion/submission to linux distros,
    • bug reporting for release candidates and mixed source releases
    • removing components
Retrieved from "https://wiki.nbi.ku.dk/w/index.php?title=Release_management&oldid=7145"