The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:DRBSD-7: The 7th International Workshop on Data Analysis and Reduction for Big Scientific Data
Authors: Sajal Sajal, Junqi Yin, Mallikarjun Shankar, and Feiyi Wang (Oak Ridge National Laboratory (ORNL)) and Wu-chun Feng (Virginia Tech)
Abstract: Training deep learning models incrementally on high-velocity data in a streaming setting can help us discover knowledge in a timely fashion. However, due to catastrophic forgetting, incrementally trained models increasingly perform poorly on the past data. We propose constructing and using a historical summary through random sampling, micro-clustering, and coreset computation along with new data during incremental training to mitigate catastrophic forgetting. We built a pipeline for incremental training with a historical summary for training deep learning models for streaming data. Through training an Artificial Neural Network for classification tasks on the MNIST dataset, our method recovered up to 47.9% lost accuracy due to catastrophic forgetting. The pipeline improved the training performance (PPL) by up to 26% while training a language model (RNN-LM) on the streaming WikiText2 dataset. This method also recovered classification accuracy by up to 25% while training a ResNet50 model on the streaming ImageNet dataset.