EOS productisation

Project goal

This project is focused on the evolution of CERN’s EOS large-scale storage system. The goal is to simplify the usage, installation, and maintenance of the system. In addition, we will add support for new client platforms, expand documentation, and implement new features/integration with other software packages.

R&D topic
R&D Topic 1: Data-centre technologies and infrastructures
Project coordinator(s)
Luca Mascetti
Technical team members
Elvin Sindrilaru
Collaborator liaison(s)
Gregor Molan, Ivan Arizanovic, Branko Blagojevic

Collaborators

Project background

Within the CERN IT department, a dedicated group is responsible for the operation and development of storage infrastructure. This infrastructure is used to store the physics data generated by the experiments at CERN, as well as the files of all members of personnel.

EOS is a disk-based, low-latency storage service developed at CERN. It is tailored to handle large data rates from the experiments, while also running concurrent complex production workloads. This high-performance system now provides more than 300 petabytes of raw disks.

EOS is also the key storage component behind CERNBox, CERN’s cloud-synchronisation service. This makes it possible to sync and share files on all major mobile and desktop platforms (Linux, Windows, macOS, Android, iOS), with the aim of providing offline availability to any data stored in the EOS infrastructure.

Recent progress

We are now in the third phase of this project. The team at Comtrade has been working to acquire further knowledge of EOS, with this activity carefully organised into nine separate work packages. Three Comtrade engineers also visited CERN and spent two weeks working side-by-side with members of the development and operations teams (helping to handle hardware failures, reconfigurations, software upgrades, and user support). We were then able to work together to create a set of technical documents describing the main aspects of EOS, for use by future administrators and operators.

In addition, we set up a proof-of-concept system using container technology. This shows the potential of the system to be used as a geographically distributed storage system and will serve as a demonstrator to potential future customers.

Next steps

We will continue our evolving work on EOS installation, documentation, and testing. We will prepare a dedicated document outlining “best practices” for operating EOS in large-scale environments.

An additional goal is to provide future customers with a virtual full-stack environment hosted at Comtrade. This would consist of an EOS instance enabled with the latest-generation namespace, a sync-and-share endpoint (using CERNBox), and an interactive data-analysis service (based on SWAN, the JupyterHub notebook used at CERN).

Publications

    X. Espinal, M. Lamanna, From Physics to industry: EOS outside HEP, Journal of Physics: Conference Series (2017), Vol. 898, https://doi.org/10.1088/1742-6596/898/5/052023. cern.ch/go/7XWH

Presentations

    X. Espinal, M. Lamanna, From Physics to industry: EOS outside HEP, Journal of Physics: Conference Series (2017), Vol. 898, https://doi.org/10.1088/1742-6596/898/5/052023. cern.ch/go/7XWH
    L. Mascetti, Comtrade EOS productization (23 January). Presented at CERN openlab technical workshop, Geneva, 2019. cern.ch/go/W6SQ
    G. Molan, EOS Documentation and Tesla Data Box (4 February). Presented at CERN EOS workshop, Geneva, 2019. cern.ch/go/9QbM