EOS productisation

EOS productisation

Project goal

This project is focused on the evolution of CERN’s EOS large-scale storage system. The goal is to simplify the usage, installation, and maintenance of the system. In addition, the project aims to add native support for new client platforms, expand documentation, and implement new features/integration with other software packages.

R&D topic
Data-centre technologies and infrastructures
Project coordinator(s)
Luca Mascetti
Team members
Elvin Sindrilaru
Collaborator liaison(s)
Gregor Molan, Ivan Arizanovic, Branko Blagojevic

Collaborators

Project background

Within the CERN IT department, a dedicated group is responsible for the operation and development of the storage infrastructure. This infrastructure is used to store the physics data generated by the experiments at CERN, as well as the files of all members of personnel.

EOS is a disk-based, low-latency storage service developed at CERN. It is tailored to handle large data rates from the experiments, while also running concurrent complex production workloads. This high-performance system now provides more than 300 petabytes of raw disks.

EOS is also the key storage component behind CERNBox, CERN’s cloud-synchronisation service. This makes it possible to sync and share files on all major mobile and desktop platforms (Linux, Windows, macOS, Android, iOS), with the aim of providing offline availability to any data stored in the EOS infrastructure.

Recent progress

Comtrade's team continued to acquire further knowledge of EOS, profiting from their visit to CERN and from working side-by-side with members of the development and operations teams. This helped them to improve their work on EOS installation, documentation, and testing.

In particular, a dedicated document describing best practices for operating EOS in large-scale environments was produced, as well as a full-stack virtual environment hosted at Comtrade. This shows the potential of the system when used as a geographically distributed storage system.

Next steps

The project will focus on improving and updating the EOS technical documents, for future administrators and operators. The next main goal is to host dedicated hardware resources at CERN to support prototyping of an EOS-based appliance. This will enable Comtrade to create a first version of a full storage solution and to offer it to potential customers in the future.

In addition, the team will investigate the possibility of developing a native Windows client for EOS.

Publications

    X. Espinal, M. Lamanna, From Physics to industry: EOS outside HEP, Journal of Physics: Conference Series (2017), Vol. 898, https://doi.org/10.1088/1742-6596/898/5/052023. cern.ch/go/7XWH

Presentations

    X. Espinal, M. Lamanna, From Physics to industry: EOS outside HEP, Journal of Physics: Conference Series (2017), Vol. 898, https://doi.org/10.1088/1742-6596/898/5/052023. cern.ch/go/7XWH
    L. Mascetti, Comtrade EOS productization (23 January). Presented at CERN openlab technical workshop, Geneva, 2019. cern.ch/go/W6SQ
    G. Molan, EOS Documentation and Tesla Data Box (4 February). Presented at CERN EOS workshop, Geneva, 2019. cern.ch/go/9QbM
    L. Mascetti, EOS Comtrade project (23 January). Presented at CERN openlab Technical workshop, Geneva, 2020. cern.ch/go/l9gc
    L. Mascetti, CERN Disk Storage Services (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/pF97
    G. Molan, Preparing EOS for Enterprise Users (27 January 2020). Presented at Cloud Storage Services for Synchronization and Sharing (CS3), Copenhagen, 2020. cern.ch/go/tQ7d
    G. Molan, EOS Documentation for Enterprise Users (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/swX8
    G. Molan, EOS Windows Native Client (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/P7DX
    G. Molan, EOS Storage Appliance Prototype (5 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/q8qh