Mutual information optimization for mitigating catastrophic forgetting in continual learning: An information-theoretic approach

Abstract

Continual learning systems encounter the critical challenge of catastrophic forgetting, where neural networks lose previously acquired knowledge when adapting to new tasks. In this paper, we propose Continual Mutual Information Preservation (CMIP), an information-theoretic approach that leverages Mutual Information (MI) optimization and entropy regularization to retain prior knowledge while learning compact and informative latent representations. CMIP uses an auxiliary network to estimate MI and a replay memory, in which each mini-batch comprises 50% current-task samples and 50% samples replayed from previous tasks. Experiments are conducted on the MNIST-Split and CIFAR-100-Split datasets for the class-incremental learning (Class-IL) setting. On MNIST-Split, CMIP achieves 90.97% accuracy with an 8.81% forgetting rate, outperforming EWC (20.64% accuracy, ~77% forgetting) and GEM (65.1% accuracy, ~33% forgetting). This method is applicable to real-world scenarios, such as robotic perception and real-time data streams.