Generalizing Dataset Distillation via Deep Generative Prior

George Cazenavette · Tongzhou Wang · Antonio Torralba · Alexei A. Efros · Jun-Yan Zhu

West Building Exhibit Halls ABC 356
[ Abstract ] [ Project Page ]
Tue 20 Jun 10:30 a.m. PDT — noon PDT


Dataset Distillation aims to distill an entire dataset’s knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite a recent upsurge of progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model’s latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.

Chat is not available.