Ultrafast Error-Bounded Lossy Compression for Scientific Datasets
Abstract
Today's scientific high-performance computing applications and advanced instruments are producing vast volumes of data across a wide range of domains, which impose a serious burden on data transfer and storage. Error-bounded lossy compression has been developed and widely used in the scientific community because it not only can significantly reduce the data volumes but also can strictly control the data distortion based on the user-specified error bound. Existing lossy compressors, however, cannot offer ultrafast compression speed, which is highly demanded by numerous applications or use cases (such as in-memory compression and online instrument data compression). In this paper, we propose a novel ultrafast error-bounded lossy compressor that can obtain fairly high compression performance on both CPUs and GPUs and with reasonably high compression ratios. The key contributions are threefold. (1) We propose a generic error-bounded lossy compression framework ---called SZx ---that achieves ultrafast performance through its novel design comprising only lightweight operations such as bitwise and addition/subtraction operations, while still keeping a high compression ratio. (2) We implement SZx on both CPUs and GPUs and optimize the performance according to their architectures. (3) We perform a comprehensive evaluation with six real-world production-level scientific datasets on both CPUs and GPUs. Experiments show that SZx is 2∼16x faster than the second-fastest existing error-bounded lossy compressor (either SZ or ZFP) on CPUs and GPUs, with respect to both compression and decompression.
Recommended Citation
X. Yu et al., "Ultrafast Error-Bounded Lossy Compression for Scientific Datasets," HPDC 2022 - Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing, pp. 159 - 171, Association for Computing Machinery (ACM), Jun 2022.
The definitive version is available at https://doi.org/10.1145/3502181.3531473
Department(s)
Computer Science
Keywords and Phrases
Error-Bounded Lossy Compression; Gpu; High-Speed Compressor; Scientific Data
International Standard Book Number (ISBN)
978-145039199-3
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2022, All rights reserved.
Publication Date
27 Jun 2022
Comments
This work was supported by the National Science Foundation, Grant OAC-2003709.