High-performance distributed caching technologies

Project goal

We’re exploring the suitability of a new infrastructure for key-value storage in the data-acquisition systems of particle-physics experiments. DAQDB (Data Acquisition Database) is a scalable and distributed key-value store that provides low-latency queries. It exploits Intel® Optane™ DC persistent memory, a cutting-edge non-volatile memory technology that could make it possible to decouple real-time data acquisition from asynchronous event selection

R&D topic
Computing performance and software
Project coordinator(s)
Giovanna Lehmann Miotto
Team members
Adam Abed Abud, Danilo Cicalese, Fabrice Le Goff, Remigius K Mommsen
Collaborator liaison(s)
Claudio Bellini, Aleksandra Jereczek, Grzegorz Jereczek, Jan Lisowiec, Maciej Maciejewski, Adrian Pielech, Jakub Radtke, Jakub Schmiegel, Malgorzata Szychowska, Norbert Szulc, Andrea Luiselli

Collaborators

Project background

Upgrades to the LHC mean that the data rates coming from the detectors will dramatically increase. Data will need to be buffered while waiting for systems to select interesting collision events for analysis. However, the current buffers at the readout nodes can only store a few seconds of data due to capacity constraints and the high cost of DRAM. It is therefore important to explore new, cost-effective solutions — capable of handling large amounts of data — that capitalise on emerging technologies.

Recent progress

We were able to test the first Intel Optane persistent-memory devices, enabling us to benchmark the behaviour of DAQDB on this new type of hardware. A testbed with four very powerful machines was set up, hosting Optane persistent memory and SSDs, and interconnected with a 100 Gbps network. The results are encouraging, but more work is needed to reach the performance and scalability goals required for the next generation of High-Luminosity LHC experiments (in particular ATLAS and CMS), as well as by the DUNE experiment.

Next steps

The project formally came to a close in 2019, but several developments and tests will continue in 2020. This will enable us to continue exploring how the new-storage technologies, and DAQDB, can be effectively used in data-acquisition systems.

Publications

    D. Cicalese et al., The design of a distributed key-value store for petascale hot storage in data acquisition systems. Published in EPJ Web Conf. 214, 2019.

Presentations

    M. Maciejewski, Persistent Memory based Key-Value Store for Data Acquisition Systems (25 September). Presented at IXPUG 2019 Annual Conference, Geneva, 2019. cern.ch/go/9cFB
    G. Jereczek, Let's get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage (5 November). Presented at the 4th International Conference on Computing in High-Energy and Nuclear Physics (CHEP), Adelaide, 2019. cern.ch/go/9cpL8
    J. Radtke, A Key-Value Store for Data Acquisition Systems (April). Presented at SPDK, PMDK and VTune(tm) Summit 04'19, Santa Clara, 2019. cern.ch/go/H6Rl
    G. Jereczek, The design of a distributed key-value store for petascale hot storage in data acquisition systems (12 July). Presented at 23rd International Conference on Computing in High Energy and Nuclear Physics (CHEP), Sofia, 2018. cern.ch/go/6hcX
    J. M. Maciejewski, A key-value store for Data Acquisition Systems (12 September). Presented at ATLAS TDAQ week, Cracow, 2018.
    G. Jereczek, M. Maciejewski, Data Acquisition Database (12 November). Presented at The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC18), Dallas, 2018.
    M. Maciejewski, J. Radtke, The Design of Key-Value Store for Data Acquisition Systems (5 December). Presented at NMVe Developer Days, San Diego, 2018.