Experiences with Cross-Facility Real-Time Light Source Data Analysis Workflows
Machine Learning and Artificial Intelligence
TimeFriday, 19 November 202111:20am - 11:40am CST
DescriptionWe are seeing a growth in scientific data from experimental and observational facilities that are resulting in significant new computational patterns and needs. For example, scientists running experiments at light Sources, often analyses workflows require near real-time access to compute resources in order to obtain results used for re-configuring on-going experiments. These workflows often have requirements that are different from the traditional large-scale parallel applications that have traditionally run at HPC centers. In this presentation, we present our experiences supporting two light source data analysis workflows that run on HPC resources at National Energy Research Scientific Computing Center. We discuss the characteristics of workflows, runtime requirements and associated execution challenges when running on HPC environments. We present a discussion that address execution challenges and current and future solutions for leveraging HPC resources for near real-time data analysis.