The Varroa destructor mite poses a critical threat to global honey bee populations, contributing to colony collapse through parasitic infestation. Current detection methods rely on labor-intensive visual inspections, which are prone to human error and impractical for large-scale monitoring. To address this, we present Vit4V, a novel deep learning framework that leverages temporal information from video sequences to automate accurate Varroa detection. Unlike image-based approaches, Vit4V processes short video clips of forager bees, capturing subtle visual cues and mitigating challenges such as motion blur and occlusions. By aggregating predictions across multiple clips, our method achieves robust detection with 0.986 accuracy. We also introduce the Varroa Destructor Video Dataset, a real-world dataset comprising 607 videos of infected and healthy bees, collected in an active beehive using an embedded camera system. The dataset, designed in collaboration with entomologists, is publicly released to facilitate further research. Experimental results demonstrate the capability of our approach, which offers a scalable, non-invasive solution for early infestation detection. Vit4V can potentially reduce reliance on harmful chemical treatments, empowering beekeepers to implement timely interventions and reduce colony losses. Our work bridges a critical gap in precision beekeeping, providing a practical tool with implications for ecological sustainability.