From Beats to Scores: A Multi-Modal Framework for Comprehensive Figure Skating Assessment

Accurate quantitative evaluation of the Technical Elements Score (TES) and Program Components Score (PCS) in figure skating requires exceptional skill and professionalism, making it a highly challenging task. It requires not only meticulous observation of athletes' technical movements but also the ability to appreciate and assess artistic elements according to the current scoring criteria. The multi-modal large language model (MLLM) is rapidly advancing in its ability to understand various forms of information, such as images, videos, and audio. Previous research efforts have mainly utilized audio and video in isolation, but effective evaluation in figure skating requires a unified approach. In our work, we developed a multi-modal method that first identifies sub-movements through audio-guided videos by leveraging the phenomena of "hit the beat" and integrates audio and video features specifically tailored for figure skating. This integration enhances the coordination between these modalities. Building upon this foundation, we propose a comprehensive assessment model that utilizes multi-modal series representation learning to derive TES and PCS scores and generate text-based competition evaluations based on video, audio, and contextual prompts. Extensive experiments have proven that our proposed method has state-of-the-art scoring ability and generalization performance. Our code is available at https://github.com/ycwfs/Figure-Skating-Quality-Assessment.