Mitigating Catastrophic Forgetting in Deep Learning in a Streaming Setting Using Historical Summary
SessionDRBSD-7: The 7th International Workshop on Data Analysis and Reduction for Big Scientific Data
Event Type
Workshop
Online Only
Applications
Big Data
Data Analytics
Data Management
W
TimeSunday, 14 November 202111:30am - 12pm CST
LocationOnline
DescriptionTraining deep learning models incrementally on high-velocity data in a streaming setting can help us discover knowledge in a timely fashion. However, due to catastrophic forgetting, incrementally trained models increasingly perform poorly on the past data. We propose constructing and using a historical summary through random sampling, micro-clustering, and coreset computation along with new data during incremental training to mitigate catastrophic forgetting. We built a pipeline for incremental training with a historical summary for training deep learning models for streaming data. Through training an Artificial Neural Network for classification tasks on the MNIST dataset, our method recovered up to 47.9% lost accuracy due to catastrophic forgetting. The pipeline improved the training performance (PPL) by up to 26% while training a language model (RNN-LM) on the streaming WikiText2 dataset. This method also recovered classification accuracy by up to 25% while training a ResNet50 model on the streaming ImageNet dataset.
