Online Data Analysis and Reduction: An Important Co-Design Motif for Extreme-Scale Computers
Abstract
A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.
Recommended Citation
I. Foster and M. Ainsworth and J. Bessac and F. Cappello and J. Choi and S. Di and Z. Di and A. Gok and H. Guo and X. Liang and For full list of authors, see publisher's website., "Online Data Analysis and Reduction: An Important Co-Design Motif for Extreme-Scale Computers," International Journal of High Performance Computing Applications, SAGE Publications, Jan 2021.
The definitive version is available at https://doi.org/10.1177/10943420211023549
Department(s)
Computer Science
Keywords and Phrases
Data Analysis; Exascale Computing; in Situ; Online Data Analysis and Reduction
International Standard Serial Number (ISSN)
1094-3420; 1741-2846
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2021 SAGE Publications, All rights reserved.
Publication Date
01 Jan 2021
