Release management
From NorduGrid
This page collects general information concerning the release procedure of ARC software.
Contents |
Background
- An ARC release is defined as a set of source packages AND the corresponding binaries packages on the supported platforms.
- ARC middleware has been used in production since May 2002
- ARC development is coordinated by the NorduGrid Collaboration
- ARC is developed by several projects and individual players; contributions are from a diverse, active, inhomogeneous community.
- ARC serves the user communities of many scientific domains all over the world.
- ARC releases have been created with the strong emphasis on multiplatform support, Linux distribution independence and recently also extended OS independence (Linux, Windows, Mac, Solaris), non-intrusiveness and ease of operation.
Actors
- system architects: higher level thinkers, key developers with full overview of system components
- developers: working for different projects at different locations, diverse background, different level of engagement
- core: full overview of a certain crucial part of the software
- component developer: responsible for a certain module
- task developer: shorter term assignment, most of the time student projects
- contributor: someone providing patches
- system administrators: end of the deployment chain guys with real knowledge of the software
- infrastructure decision makers: high-level actors influencing software deployment on infrastructure
- users:
- organized as larger communities ("big VO")
- individual users ("small guys")
- code keeper: the source repository maintainer, a person responsible for svn on a high level, consistency of the source code environment
- release manager: a person responsible for the complete lifecycle of a major release including eventual minor subreleases. appointed by coordination group
- bugzilla manager: a person responsible for creating relevant tags in Bugzilla
- binarians: group of developers responsible for the binary build, software distribution, packaging, binary repositories, download areas
- test team: defines and implements test, maintains test infrastructure, coordinates testing activities, executes the tests
- Technical Coordination Group (NTCG):
- middle and longer term strategies
- coordination and monitoring
- includes architects, developers, user representatives, decision makers
Development environment
- code is kept in svn spread over several svn trees, ARC has a non-monolitic source code tree (svn.nordugrid.org)
- SVN read access is provided for everyone
- SVN commit access is regulated and the commit right is granted by the the coordination group
- entry-level access is given to a dedicated area called "workarea". This is the place for initial code uploads.
- feature branches are created for implementing larger-scale new functionality and commit access is granted to assigned developers
- trunk is kept for main synchronized development and comes with a more restricted access policy
- release branches represent the most stable areas and commits to the branches are closely monitored, towards the end of the release procedure even controlled by the release coordinator
- Bugzilla is used to report software defects and to act upon them
- feature requests are also collected in Bugzilla
- bugs, including feature requests are periodically reviewed and prioritized by the core group
- WIKI is used as a workarea
- to collect development plans, technical drafts, ideas
- release coordination
- configuration templates
- quick guides, HOWTOs
- nordugrid-discuss mailing
- high traffic technical list
- Ticketing system
- an issue tracking system is used to provide user support
- eventual software defects discovered during the user assistance process is recorded to the bugzilla by the support staff
- There are two different nightly build systems: A nightly "standard" build (Kosice http://vls.grid.upjs.sk/testing/index.html) which is in fact running the revision tests thus usually the last revision from a given day corresponds to the nightly build and a nightly "package" build (Copenhagen http://download.nordugrid.org/builds/). The standard build does each build step separately while the package build does everything from tarball/src.rpm - including creating pristine environments for each package build. This system is identical to the release builds. So in summary the nightly "package" build really checks the whole build and package chain for release packagesARC Build and testing systems: currently there are two production systems in use to perform.
- automatic nightly builds of several svn areas
- producing binary packages for tagged releases
Release categories
- major release
- longer term planning
- 3-6 months preparation
- alpha, beta, rcx test releases
- introduces new components, features
- obsoletes components
- may break backward compatibility
- minor release
- planned bugfix release
- max one month preparation
- no new features, no new components
- only fixes
- scheduled release
- emergency release
- unplanned urgent release to fix a security issue or a critical bug
- maximum two weeks preparation
- binary update of a release
- sometimes due to a change in the external dependencies ARC releases have to be rebuilt.
- This "semi release" does not affect the ARC source code, no re-taging takes place, only new binaries are built
- can happen for both major, minor or emergency releases
Workflow
Scheduled release procedure
- Major or minor releases follow a scheduled release procedure
- The contents of major ARC releases are trying to follow a high level ARC development roadmap
- Minor releases contain only bugfixes
- Planning phase (community process)
- overview of external dependencies
- Prioritizing bugs to be fixed
- collecting requests for new features (only for major release)
- consider adding newly developed components to the code base (only for major release)
- add support for new platforms (only for major release)
- introduce changes in the packaging, software distribution (only for major release)
- Decision phase (carried out by the NTCG)
- decision takes into account the status (maturity) of release candidate components
- the release content is defined in terms of components and bugs to be fixed
- supported platforms decided
- time planning of the release preparation
- appointing release coordinator testing team, binarian
- Preparation of the release (release coordinator, developer and testing team)
- Creation of the release branch
- Documentation updates, Release notes
- Release candidate(s) are created from the branch
- Testing
- Release candidate is deployed on the test infrastructure
- The testing team executes all the test suits
- Larger testing campaigns are organized using the release candidate (involving users and sysadmins outside the testing team).
- Release is ready if
- All the critical bugs recorded in the bugzilla are closed
- Functionality decided in planning phase is succesfully tested on the release candidate
- Binary packages are available for all supported platforms
- Documentation reflecting changes from the previous release is in place
- Relevant Bugzilla tags are in place
Emergency release procedure
- running into a MAJOR ISSUE: a software defect considered critical or a security whole was discovered during
- everyday usage scenario
- bug fixing, debuging
- regular testing
- triggered by external dependency
- the reporting of the MAJOR ISSUE can happen through several channels: bugzilla, mailing list, direct contact to core coordination group members
- The decision whether ISSUE requires a emergency release is taken by the nordugrid technical coordination group (NTCG). Options:
- prepare an emergency release
- prepare only a patch offering partial solution only for some of the platforms
- don't prepare an emergency release but fix the issue with the next scheduled (minor/major) release
- The emergence release is being prepared:
- release coordinator appointed by NTCG
- identify target branch and create a emergency release branch out of that, decide upon the release number
- investigate the problem, create a strategy to resolve the issue
- once the fix is available the NTCG appoints an independent tester
- after successful testing the release coordinator completes the process by
- updating release notes reflecting changes (fix or workaround)
- creating tag (deadline should be announced to developers), packages builds
- populating repositories
- Emergency release is ready
- the entire release procedure is closely controlled by the release coordinator
- the process can't take more time than two weeks
Policies
- ARC packages pushed to distribution repositories must always be certified and taken from final releases (e.g. no release candidates are allowed).
placeholder
- these are not yet properly covered:
- testing,
- external software,
- integration,
- rollout tests,
- inclusion/submission to linux distros,
- bug reporting for release candidates and mixed source releases
