Efficient Lossy Compression for Scientific Data based on Pointwise Relative Error Bound
Abstract
An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualities on different types of data sets respectively. The compression ratio of our lossy compressor is higher than that of other state-of-the-art compressors by 17.2-618 percent on the climate simulation data and 30-210 percent on the N-body simulation data, with the same relative error bound and without degradation of the overall visualization effect of the entire data.
Recommended Citation
S. Di et al., "Efficient Lossy Compression for Scientific Data based on Pointwise Relative Error Bound," IEEE Transactions on Parallel and Distributed Systems, vol. 30, no. 2, pp. 331 - 345, Institute of Electrical and Electronics Engineers (IEEE), Feb 2019.
The definitive version is available at https://doi.org/10.1109/TPDS.2018.2859932
Department(s)
Computer Science
Keywords and Phrases
High Performance Computing; Lossy Compression; Relative Error Bound; Science Data
International Standard Serial Number (ISSN)
1045-9219; 1558-2183
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
01 Feb 2019
Comments
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC.