Toward Aggregated Asynchronous Checkpointing
Event Type
Workshop
Parallel Programming Languages and Models
Reliability and Resiliency
W
TimeMonday, 15 November 202110:30am - 11am CST
Location227
DescriptionHigh-Performance Computing (HPC) applications need to check-point massive data sizes at scale with increasing frequency. Multi-level asynchronous checkpoint runtimes like VELOC (Very Low Overhead Checkpoint Strategy) are gaining popularity among application scientists for their ability to leverage fast node-local storage and flush independently to stable, external storage (e.g., parallel file systems) in the background. Currently, VELOC adopts a one-file-per-process flush strategy, which results in a large number of files being written to external storage, thereby overwhelming metadata servers and making it difficult to transfer and access checkpoints as a whole. This paper discusses the challenges and opportunities of designing aggregation techniques for asynchronous multi-level checkpointing. To this end, we implement and studied two aggregation strategy, study their limitations and propose a new aggregation strategy specifically for asynchronous multi-level checkpointing.
Author/Presenter
