Toward Scalable Data Processing in Python with CLIPPy

Workshop:IA^3 2021: 11th Workshops on Irregular Applications: Architectures and Algorithms

Authors: Peter Pirkelbauer (Lawrence Livermore National Laboratory, University of Central Florida) and Seth Bromberger, Keita Iwabuchi, and Roger Pearce (Lawrence Livermore National Laboratory)

Abstract: The Python programming language has become a popular choice for data scientists. While easy to use, the Python language is not well suited to drive data science on large scale systems.

This paper presents a first prototype of CLIPPy (Command line interface plus Python), a user-side interface in Python that connects to high-performance computing environments with non-volatile memory. CLIPPy queries available executable files and prepares a Python API on the fly. The executables offer an interface to a backend that can execute on large-scale systems. The executables can be implemented in any language, for example C++ . CLIPPy and the executables are loosely coupled and communicate through a JSON based interface.

The underlying philosophy, design challenges, and a prototype implementation that accesses data stored in non-volatile memory will be discussed.

Website:

Back to IA^3 2021: 11th Workshops on Irregular Applications: Architectures and Algorithms Archive Listing

Back to Full Workshop Archive Listing