The Service Analysis and Network Diagnosis Data Pipeline

Workshop:INDIS'21: 8th Workshop on Innovating the Network for Data-Intensive Science

Authors: Derek Weitzel (University of Nebraska, Lincoln); Shawn McKee (University of Michigan); Brian Bockelman (Morgridge Institute of Research); John Thiltges (University of Nebraska-Lincoln); Marian Babik (European Organization for Nuclear Research (CERN)); and Ilija Vukotic (University of Chicago)

Abstract: Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements into a message bus. The message bus allows the project to send the data to multiple consumers, including a tape archive, an Elasticsearch database, and a peer infrastructure at CERN. In this paper, we explain the architecture and evolution of the SAND data pipeline, the scale of the resulting dataset, and how it supports a wide variety of network analysis applications.

Back to INDIS'21: 8th Workshop on Innovating the Network for Data-Intensive Science Archive Listing

Back to Full Workshop Archive Listing