Abstract

In today’s extreme-scale scientific simulations, vast volumes of data are being produced such that the data cannot be accommodated by the parallel file system or the data writing/ reading performance will be fairly low because of limited I/O bandwidth. In the past decade, many snapshot-based (or space-based) lossy compressors have been developed, most of which rely on the smoothness of the data in space. However, the simulation data may get more and more complicated in space over time steps, such that the compression ratios decrease significantly. In this paper, we propose a novel, hybrid lossy compression method by leveraging spatiotemporal decimation under the SZ compression model. The contribution is twofold. (1) We explore several strategies of combining the decimation method with the SZ lossy compression model in both the space dimension and time dimension during the simulation. (2) We investigate the best-fit combined strategy upon different demands based on a couple of typical real-world simulations with multiple fields. Experiments show that either the space-based SZ or time-based SZ leads to the best rate distortion. Decimation methods have very high compression rate with low rate distortion though, and SZ combined with temporal decimation is a good tradeoff.

Meeting Name

2020 IEEE/ACM 4th International Workshop on Data Reduction for Big Scientific Data, DRBSD-4 @SC'18 (2018: Nov. 11-16, Dallas, TX)

Department(s)

Computer Science

Document Type

Article - Conference proceedings

Document Version

Final Version

File Type

text

Language(s)

English

Rights

© 2018 Association for Computing Machinery (ACM), All rights reserved.

Publication Date

16 Nov 2018

Download

Share

 
COinS