Lightning Talk: DynamicDeepFlow - Classification of Network Traffic Flow Changes using Unsupervised Learning

Workshop:INDIS'21: 8th Workshop on Innovating the Network for Data-Intensive Science

Authors: Sheng Shen (Lawrence Berkeley National Laboratory (LBNL))

Abstract: Understanding flows in network traffic has great importance in designing and building robust networking infrastructure. Recent efforts via industry and research are developing monitoring tools to capture real-time flow data, predict future traffic and mirror packet headers. However, all of these require offline analysis of the data to understand the big versus small flows, impact on devices and recognize congestion hot spots. With growing network complexities, there is a need for these monitoring systems to learn in real-time and identify anomalies. Particularly with large file transfers in Internet setups, identifying new sites, unusual login activities and traffic spikes can help reduce network downtimes. In this paper, we target the challenge of flow behavior changes on the network by developing an innovative unsupervised deep learning model, DynamicDeepFlow, that can recognize new flow patterns in real-time. Trained on network snapshots over two years, the proposed approach leverages a combination of deep learning model, autoencoder, and shallow learning model, k-means++ to recognize real traffic change patterns. We apply this study to real world data from Energy Sciences Network designed to support U.S. scientific research. Our approach explicitly learns the dynamic spatiotemporal traffic patterns, showing when tested on data from 2018-2020, specific days and nights when the network experience new flows that were never seen before. DynamicDeepFlow is a first of its kind real-time network monitoring system that can aid in network operations and used in conjunction with current flow data collected.

Back to INDIS'21: 8th Workshop on Innovating the Network for Data-Intensive Science Archive Listing

Back to Full Workshop Archive Listing