We are working with Rackspace and the open-source community to enhance OpenStack’s container support for use in high-energy physics. CERN has been using OpenStack since 2013 to provide virtual-machine resources. We are now adding containers to the standard production service; this enables rapid prototyping of applications and scaling of these services as they grow within the common accounting and resource pledges agreed with the WLCG.
Container technologies are an area of great interest, both across the ICT industry and within the physics community. Application developers are looking for more flexible ways of delivering components based on micro-services, such as SWAN (a platform created at CERN for performing interactive data analysis in the cloud) and Jupyter notebooks (an open-source web application that enables users to create and share documents containing live code, equations, visualisations, and narrative text).
During 2017, OpenStack Magnum deployments grew significantly: they now represent 14% of all deployments. The CERN collaboration with Rackspace contributed over 60 updates to Magnum, including significant improvements related to cluster monitoring, online upgrades, and heterogeneous clusters of different node sizes.
One area of focus in 2017 was working on scaling large Kubernetes clouds; this was important for investigations into the future design of the ATLAS experiment’s upgraded high-level trigger system for Run 4 of the LHC. This work required us to make improvements to both Magnum itself and the underlying orchestration component.
Given these significant contributions, the CERN openlab fellow working on the project has recently been elected to lead the Magnum project team. This is an international role with responsibility for coordinating development plans, as well as overseeing the next release of Magnum.
The current phase of our collaboration with Rackspace comes to a close at the start of 2018. The container service is now in production in over 100 clusters, and the community will continue to develop functionality. Following the success of this project, further areas for collaboration with Rackspace are being explored.
- S. Trigazis, OpenStack @ CERN: Status update (4 February), Presented at CentOS dojo, Brussels, 2017. cern.ch/go/kh8v
- S. Trigazis, OpenStack Magnum Pike and the CERN cloud (28 February), Presented at Containers Meetup, Paris, 2017. cern.ch/go/8BRs
- S. Trigazis, From swarm to swarm-mode in the CERN container service (6 April), Presented at Docker Meetup, Zurich, 2017. cern.ch/go/dlt7
- R. Rocha, CERN Container experience (5 May), Presented at OpenStack Summit, Boston, 2017. cern.ch/go/8MTf
- S. Trigazis, OpenStack/Magnum and the CERN container service (28 June), Presented at German OpenStack Days 2017, Munich, 2017. cern.ch/go/Nq7c
- S. Oak, Cloud Log Data Analysis (11 August), Presented at CERN openlab summer students’ lightning talks, Geneva, 2017. cern.ch/go/FD8T
- M. Sommer, Stateful services in containers (15 August), Presented at CERN openlab summer students’ lightning talks, Geneva, 2017. cern.ch/go/Gj7Z
- C. F. Souza, Kubernetes as a batch scheduler (and more...) (15 August), Presented at CERN openlab summer students’ lightning talks, Geneva, 2017. cern.ch/go/T7JL
- S. Trigazis, Rackspace Containers at Scale (21 September), Presented at CERN openlab Open Day, Geneva, 2017. cern.ch/go/9Lll
- T. Bell, Clouds at CERN (26 September), Presented at OpenStack days UK, London, 2017. cern.ch/go/v87D