Presentation
P13 - A Distributed LogStore Design with Multi-Reader, Multi-Writer Semantics for Streaming Applications
Presenter
DescriptionIn this work we describe the design and implementation of a distributed logstore that can be used for storing events from streaming applications such as Telemetry and Satellite Remote Sensing. The logstore provides multi-writer, multi-reader (MWMR) semantics. It also totally orders events using timestamps as keys. Our implementation uses a distributed clock synchronization algorithm to synchronize all the processes on a cluster with respect to a master process. Since the logstore is designed to support streaming applications which run for long duration and sample data at constant rates, we used two levels of buffers in our implementation to reduce the total number of disk accesses. Events are buffered in CPU memories and NVMe files before eventually reaching disks. Timer threads running in the background control the flushing of data between memories and disks. They also handle the memory management in the system, thereby making it possible to stream several gigabytes of data over long periods of time. The logstore implementation is hybrid(multi-process and multi-threaded). We used multi-threaded RPCs, MPI, Pthreads and Argobots for implementation. All the IO in the logstore is performed using Parallel HDF5. We also implemented a KeyValueStore interface to the logstore for Client applications.
TimeMonday, June 319:02 - 19:04 CEST
LocationHG F 30 Audi Max
Event Type
Poster
