Improving Performance of Data Dumping with Lossy Compression for Scientific Simulation
Abstract
Because of the ever-increasing data being produced by today's high performance computing (HPC) scientific simulations, I/O performance is becoming a significant bottleneck for their executions. An efficient error-controlled lossy compressor is a promising solution to significantly reduce data writing time for scientific simulations running on supercomputers. In this paper, we explore how to optimize the data dumping performance for scientific simulation by leveraging error-bounded lossy compression techniques. The contributions of the paper are threefold. (1) We propose a novel I/O performance profiling model that can effectively represent the I/O performance with different execution scales and data sizes, and optimize the estimation accuracy of data dumping performance using least square method. (2) We develop an adaptive lossy compression framework that can select the bestfit compressor (between two leading lossy compressors SZ and ZFP) with optimized parameter settings with respect to overall data dumping performance. (3) We evaluate our adaptive lossy compression framework with up to 32k cores on a supercomputer facilitated with fast I/O systems and using real-world scientific simulation datasets. Experiments show that our solution can mostly always lead the data dumping performance to the optimal level with very accurate selection of the bestfit lossy compressor and settings. The data dumping performance can be improved by up to 27% at different scales.
Recommended Citation
X. Liang et al., "Improving Performance of Data Dumping with Lossy Compression for Scientific Simulation," Proceedings of the IEEE International Conference on Cluster Computing, Institute of Electrical and Electronics Engineers (IEEE), Sep 2019.
The definitive version is available at https://doi.org/10.1109/CLUSTER.2019.8891037
Meeting Name
2019 IEEE International Conference on Cluster Computing, ICCC (2019: Sep. 23-26, Albuquerque, NM)
Department(s)
Computer Science
International Standard Book Number (ISBN)
978-172814734-5
International Standard Serial Number (ISSN)
1552-5244
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
01 Sep 2019
Comments
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC. This research is also supported by NSF Award No. 1513201.