Abstract
Todays exa-scale scientific applications or advanced instruments are producing vast volumes of data, which need to be shared/transferred through the network/devices with relatively low bandwidth (e.g., data sharing on WAN or transferring from edge devices to supercomputers). Lossy compression is one of the candidate strategies to address the big data issue. However, little work was done to make it resilient against silent errors, which may happen during the stage of compression or data transferring. In this paper, we propose a resilient error-bounded lossy compressor based on the SZ compression framework. Specifically, we design a new independentblock-wise model that decomposes the entire dataset into many independent sub-blocks to compress then, we design and implement a series of error detection/correction strategies elaboratively for each stage of SZ. Our method is arguably the first algorithmbased fault tolerance (ABFT) solution for lossy compression. Our proposed solution incurs negligible execution overhead in the faultfree situation. Upon soft errors happening, it ensures decompressed data strictly bounded within users requirement with a very limited degradation of compression ratio and low overhead.
Recommended Citation
S. Li et al., "Resilient Error-Bounded Lossy Compressor for Data Transfer," Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2021, St. Louis, MO), Association for Computing Machinery (ACM), Nov 2021.
The definitive version is available at https://doi.org/10.1145/3458817.3476195
Meeting Name
International Conference for High Performance Computing, Networking, Storage and Analysis, SC'21 (2021: Nov. 14-19, St. Louis, MO)
Department(s)
Computer Science
Keywords and Phrases
Algorithm Based Fault Tolerance; Data transfer; Lossy compression
International Standard Book Number (ISBN)
978-145038442-1
International Standard Serial Number (ISSN)
2167-4337; 2167-4329
Document Type
Article - Conference proceedings
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2021 The Authors, All rights reserved.
Creative Commons Licensing
This work is licensed under a Creative Commons Attribution 4.0 License.
Publication Date
19 Nov 2021
Comments
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1617488, Grant No. 1619253 and Grant No. 2003709.