SC21 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Software APIs for Programming Exascale HPC

Authors: Cory Snavely (National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory (LBNL)), Maxime Martinasso (Swiss National Supercomputing Centre (CSCS))

Abstract: This BoF will provide a forum to discuss the emergence of software application programming interfaces (APIs) for interacting with (pre-)exascale HPC resources. As automation is increasingly required for complex workflows, and science gateways become an increasingly popular digital interface for user communities, it is important for HPC centers to provide a modern and secure means for software to interface with batch workload managers, storage systems, and other supercomputing facility resources. The session will present overviews of various API initiatives, describe their application in software projects, and answer attendee questions about using and implementing APIs for HPC projects.

Long Description: The goal of this BoF is to discuss the need for new methods to interact with (pre-)exascale HPC resources and form a community among HPC providers and users to foster the development of modern tools for building scientific portals and integrating automated workflows with HPC resources. Today, HPC provides bespoke solutions for resource access, and scientific portals and workflows use custom integration mechanisms based on legacy technology such as SSH. Meanwhile, API-based architectures have become ubiquitous for integration work and offer a secure, efficient and standardized technology that can also be applied to create scientific portals or automate workflows with different service providers. This BoF will engage HPC users and providers to discuss the benefits, requirements and feasibility of interacting with HPC systems using APIs.

Supercomputing is an ideal venue for this BoF as it brings major HPC resource providers together with scientific computing users, providing an opportunity to discuss existing API offerings of several international HPC centers, stimulate conversation on implementation and adoption, and steer future efforts.

The BoF will present a panel of experts from facilities that provide API capabilities: CSCS, NERSC, OLCF, and TACC, along with Globus, a well-known data transfer service. The panel will be complemented by lightning talks highlighting representative scientific projects to illustrate typical applications of APIs in scientific research. Two talks will describe real-world examples of distributed workflows that perform data movement and automated orchestration of computational tasks, and the third will demonstrate how scientific portals can be built easily and efficiently with Python using an API framework.

The audience will engage with live polling and submission / up-voting of panel questions, which will be seeded. The discussion will both HPC providers and users.

Example live-poll questions:

* Providers:

- Do you intend to build an API for your facility?

- What are the biggest challenges to deploying an API for your facility?

* Users:

- If APIs were available for HPC resources, would you use them?

- What API features would you need to automate your workflow?

Example questions addressing implementation:

- What resources and functions of an HPC system can be programmable?

- What is involved in deploying an API for an HPC provider?

- How is authentication and authorization handled?

- How is data movement between facilities handled?

- Is it secure to open HPC resources to API access?

- How do you ensure high availability of the API?

Example questions addressing adoption:

- As a software developer, how would you use an HPC API?

- What would motivate you to retool your workflow?

- What are the challenges in creating scientific portals that access HPC resources?

- What other API or related technologies are relevant to this discussion?

- How portable is an API-based workflow across HPC resource providers?

The result of this BoF will be a package of materials: panel member and lightning talk presenter contact info, slides, live-poll questions and responses, and a summary of the panel discussion. During live-polling, we will allow participants to supply contact information and indicate their interest in joining a list to facilitate networking after the conference.


Back to Birds of a Feather Archive Listing