Experiences with Cross-Facility Real-Time Light Source Data Analysis Workflows

Workshop:Urgent HPC: HPC for Urgent Decision Making

Authors: Anna Giannakou, Johannes Blaschke, Deborah Bard, and Lavanya Ramakrishnan (Lawrence Berkeley National Laboratory (LBNL))

Abstract: We are seeing a growth in scientific data from experimental and observational facilities that are resulting in significant new computational patterns and needs. For example, scientists running experiments at light Sources, often analyses workflows require near real-time access to compute resources in order to obtain results used for re-configuring on-going experiments. These workflows often have requirements that are different from the traditional large-scale parallel applications that have traditionally run at HPC centers. In this paper, we present our experiences supporting two light source data analysis workflows that run on HPC resources at National Energy Research Scientific Computing Center. We discuss the characteristics of workflows, runtime requirements and associated execution challenges when running on HPC environments. We present a discussion that address execution challenges and current and future solutions for leveraging HPC resources for near real-time data analysis.

Back to Urgent HPC: HPC for Urgent Decision Making Archive Listing

Back to Full Workshop Archive Listing