In today’s extreme-scale scientific simulations, vast volumes of data are being produced such that the data cannot be accommodated by the parallel file system or the data writing/ reading performance will be fairly low because of limited I/O bandwidth. In the past decade, many snapshot-based (or space-based) lossy compressors have been developed, most of which rely on the smoothness of the data in space. However, the simulation data may get more and more complicated in space over time steps, such that the compression ratios decrease significantly. In this paper, we propose a novel, hybrid lossy compression method by leveraging spatiotemporal decimation under the SZ compression model. The contribution is twofold. (1) We explore several strategies of combining the decimation method with the SZ lossy compression model in both the space dimension and time dimension during the simulation. (2) We investigate the best-fit combined strategy upon different demands based on a couple of typical real-world simulations with multiple fields. Experiments show that either the space-based SZ or time-based SZ leads to the best rate distortion. Decimation methods have very high compression rate with low rate distortion though, and SZ combined with temporal decimation is a good tradeoff.
X. Liang et al., "Exploring Best Lossy Compression Strategy by Combining SZ with Spatiotemporal Decimation," Proceedings of the 2020 IEEE/ACM 4th International Workshop on Data Reduction for Big Scientific Data (2018, Dallas, TX), Association for Computing Machinery (ACM), Nov 2018.
2020 IEEE/ACM 4th International Workshop on Data Reduction for Big Scientific Data, DRBSD-4 @SC'18 (2018: Nov. 11-16, Dallas, TX)
Article - Conference proceedings
© 2018 Association for Computing Machinery (ACM), All rights reserved.
16 Nov 2018