MSU-4S - The Michigan State University Four Seasons Dataset

Public datasets, such as KITTI, nuScenes, and Waymo, have played a key role in the research and development of autonomous vehicles and advanced driver assistance systems. However, many of these datasets fail to incorporate a full range of driving conditions; some datasets only contain clear-weather conditions, underrepresenting or entirely missing colder weather conditions such as snow or autumn scenes with bright colorful foliage. In this paper, we present the Michigan State University Four Seasons (MSU-4S) Dataset, which contains real-world collections of autonomous vehicle data from varied types of driving scenarios. These scenarios were recorded throughout a full range of seasons, and capture clear, rainy, snowy, and fall weather conditions, at varying times of day. MSU-4S contains more than 100,000 two- and three-dimensional frames for camera, lidar, and radar data, as well as Global Navigation Satellite System (GNSS), wheel speed, and steering data, all annotated with weather, time-of-day, and time-of-year. Our data includes cluttered scenes that have large numbers of vehicles and pedestrians; and it also captures industrial scenes, busy traffic thoroughfare with traffic lights and numerous signs, and scenes with dense foliage. While providing a diverse set of scenes, our data incorporate an important feature: virtually every scene and its corresponding lidar, camera, and radar frames were captured in four different seasons, enabling unparalleled object detection analysis and testing of the domain shift problem across weather conditions. In that context, we present detailed analyses for 3D and 2D object detection showing a strong domain shift effect among MSU-4S data segments collected across different conditions. MSU-4S will also enable advanced multimodal fusion research including different combinations of camera-lidar-radar fusion, which continues to be of strong interest for the computer vision, autonomous driving and ADAS development communities. The MSU-4S dataset is available online at https://egr.msu.edu/waves/msu4s.