Robust AD: A Real World Benchmark Dataset For Robustness in Industrial Anomaly Detection

Successful deployment of anomaly detection algorithms in real-world industrial settings, particularly for defect detection tasks, is often hampered by performance degradation due to discrepancies between the training environment and deployment conditions. These discrepancies, known as domain shifts, can encompass variations in lighting, object pose, background clutter, and other factors. Existing benchmarks for measuring robustness of anomaly detection rely on synthetic transformations that often fail to capture the complexities of real-world variations. We address this gap by proposing a novel benchmark, RobustAD, specifically designed to evaluate the robustness of anomaly detection models in real-world scenarios. RobustAD features a curated dataset of defect detection images with meticulously controlled distribution shifts across multiple dimensions relevant to practical applications and more closely mirrors real-world deployment scenarios. Furthermore, we introduce a novel metric, "Average Relative Drop," which measures the relative performance drop in target domains with domain shifts, compared to the training domain. This practitioner-centric metric provides a more meaningful evaluation of a model's robustness in real-world scenarios. By leveraging RobustAD and the Average Relative Drop metric, researchers can evaluate their model against real-world domain shifts, under diverse real-world conditions, providing a more accurate assessment of model's suitability for practical deployment. We evaluate state-of-the-art models on RobustAD across a spectrum of tasks establishing a comprehensive benchmark for assessing real-world robustness in industrial anomaly detection. The dataset is available at https://huggingface.co/datasets/AmazonScience/RobustAD