With the increasing number of incidents in complex railroad environments, there is an urgent need for automated systems that can predict dangerous events through accurate detection and distance measurement between trains and various hazards. To address these challenges, we introduce Near-Miss Detector (NMD), an integrated framework that leverages specialized models to accurately detect possible collisions between trains and people or objects in the railroad environment via monocular cameras installed in front of a train. NMD constructs a comprehensive, 3-dimensional view of a given scene via object detection, instance segmentation and depth estimation. In this view, the risk of an accident is measured through the distance between the moving train and detected objects in the scene. In order to apply NMD in real-world scenarios, we present a novel depth-calibration mechanism based on constant geometrical properties of a railroad environment, such as the gauge of the rail track. To validate our work, we collected a dataset of measurements from multiple train stations in order to accurately represent the diversity and challenges of a complex railroad environment. NMD demonstrates robust performance in object detection, track segmentation, and distance measurement while maintaining suitable processing latency. This work contributes to the field of automated railway safety monitoring by showing the feasibility of monocular vision-based distance measurement in complex railway environments, offering a cost-effective solution for improving railway safety systems.