CVPR Conference (2023)
Qiao Gu(University of Toronto), Dongsub Shim, Florian Shkurti(University of Toronto)
Abstract
Catastrophic forgetting has been a major challenge in continual learning, where the model needs to learn new tasks with limited or no access to data from previously seen tasks. To tackle this challenge, methods based on knowledge distillation in feature space have been proposed and shown to reduce forgetting. However, most feature distillation methods directly constrain the new features to match the old ones, overlooking the need for plasticity. To achieve a better stability-plasticity trade-off, we propose Backward Feature Projection (BFP), a method for continual learning that allows the new features to evolve up to a learnable linear transformation of the old features. BFP preserves the linear separability of the old classes while allowing new feature directions to emerge to accommodate new classes. Experiments show that BFP can be integrated with existing experience replay methods and boost performance by a significant margin. We also demonstrate that BFP helps learn a better representation space, in which linear separability is well preserved during CL and linear probing can recover high accuracies.