The Surprising Utility of Group Partitioning in Improving Conformal Prediction of Visual Classifiers under Distributional Shifts

Conformal prediction (CP) provides model-agnostic uncertainty quantification (UQ) with finite-sample coverage guarantees. However, standard CP ensures marginal coverage rather than providing guarantees for individual inputs, which can lead to systematic over- or under-coverage in certain regions of the input space. This problem is exacerbated even further since models are often deployed in test scenarios where they encounter data from disparate modes (e.g., distribution shifts) of data unseen during training. To address this, Group-based CP aims to move towards conditional coverage by partitioning data into structured subgroups and provide group level guarantees. However, its adoption has been limited due to challenges in defining meaningful groups. Most approaches rely on class labels or heuristic partitions which are hard to define and even calibrate under distributional shifts. As a result, recent research has favored other strategies, shifting focus away from group-based CP. In this study, we revisit and analyze this family of methods and introduce a novel variant where groups are formed based on their alignment or likelihood with the training distribution. We systematically evaluate this approach on benchmark datasets, demonstrating that likelihood-based grouping improves coverage reliability and outperforms standard class-based methods, indicating its potential for robust uncertainty quantification under distribution shifts. Our codes including a demo can be found here: \url{https://github.com/kowshikthopalli/LikelihoodGroupConformal_Prediction/}