Optimizing Lossy Compression with Adjacent Snapshots for N-Body Simulation Data
Abstract
Today's N-body simulations are producing extremely large amounts of data. The Hardware/Hybrid Accelerated Cosmology Code (HACC), for example, may simulate trillions of particles, producing tens of petabytes of data to store in a parallel file system, according to the HACC users. In this paper, we design and implement an efficient, in situ error-bounded lossy compressor to significantly reduce the data size for N-body simulations. Not only can our compressor save significant storage space for N-body simulation researchers, but it can also improve the I/O performance considerably with limited memory and computation overhead. Our contribution is threefold. (1) We propose an efficient data compression model by leveraging the consecutiveness of the cosmological data in both space and time dimensions as well as the physical correlation across different fields. (2) We propose a lightweight, efficient alignment mechanism to align the disordered particles across adjacent snapshots in the simulation, which is a fundamental step in the whole compression procedure. We also optimize the compression quality by exploring best-fit data prediction strategies and optimizing the frequencies of the space-based compression vs. time-based compression. (3) We evaluate our compressor using both a cosmological simulation package and molecular dynamics simulation data - two major categories in the N-body simulation domain. Experiments show that under the same distortion of data, our solution produces up to 43% higher compression ratios on the velocity field and up to 300% higher on the position field than do other state-of-the-art compressors (including SZ, ZFP, NUMARCK, and decimation). With our compressor, the overall I/O time on HACC data is reduced by up to 20% compared with the second-best compressor.
Recommended Citation
S. Li et al., "Optimizing Lossy Compression with Adjacent Snapshots for N-Body Simulation Data," Proceedings of the 2018 IEEE International Conference on Big Data, Big Data 2018, pp. 428 - 437, Institute of Electrical and Electronics Engineers (IEEE), Jan 2019.
The definitive version is available at https://doi.org/10.1109/BigData.2018.8622101
Meeting Name
2018 IEEE International Conference on Big Data, Big Data 2018 (2018: Dec. 10-13, Seattle, WA)
Department(s)
Computer Science
Keywords and Phrases
Error-Bounded Lossy Compression; I/O Performance; Large Science Data; N-Body Simulation
International Standard Book Number (ISBN)
978-153865035-6
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
22 Jan 2019
Comments
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC. This research is also supported by NSF Award No. 1513201.