Hyperspectral imaging (HSI) has emerged as a groundbreaking technology for non-invasive and non-destructive food quality assessment, eliminating the limitations of traditional measurement systems. Simultaneously capturing spectral and spatial data, HSI provides detailed chemical and structural insights, enabling more precise quality evaluation. However, most existing research in the food domain primarily relies on average spectral data, failing to harness the full potential of the rich spatial and spectral information contained in hyperspectral images. To address this gap, this study introduces a novel Convolution-Attention Fusion model named "Agro-Net," designed for hyperspectral image feature extraction to enhance the accuracy of food quality prediction. The superior performance of Agro-Net, compared to state-of-the-art machine learning models in classification and regression tasks, underscores the importance of utilizing both spatial and spectral data for more effective food quality assessment.