IMC: A Benchmark for Invariant Learning under Multiple Causes

Deploying machine learning models in new environments requires robust algorithms that can handle distribution shifts. Invariant learning aims to achieve this by learning features that remain predictive across diverse environments. However, the previous benchmark dataset for invariant learning algorithms typically assumes that each class is determined by a single cause, leaving the study of scenarios where multiple causes contribute to class determination largely unexplored. To achieve a reliable performance in the real world, it should make reasonable decisions based on the available causes, even when some causes are unobserved. This necessitates learning and effectively utilizing all relevant causes. In this paper, we introduce a new benchmark designed to evaluate invariant learning in settings where multiple causes determine class labels. To enable systematic and quantitative evaluation, we construct new testbeds that allow per-cause assessment and analyze how well invariant learning algorithms capture diverse causes. Our study newly reveals that existing algorithms tend to focus disproportionately on specific causes, and we provide an in-depth analysis of this phenomenon. Our benchmark introduces six new datasets across computer vision and natural language processing (NLP) domains. Finally, we provide a simple but effective method for improving invariant learning algorithms to ensure a more comprehensive utilization of multiple causes.