Image Matching aims to establish correspondences between images, which plays a crucial role in camera pose estimation. Recent detector-free approaches have demonstrated remarkable progress in this field; however, they still suffer from low computational efficiency. The primary sources of their high time complexity are as follows: (1) the computationally expensive feature extraction backbone, and (2) the redundant feature updating process. To address these challenges, we propose an effective and efficient approach. Specifically, for issue (1), we design a lightweight backbone while preserving model performance through knowledge distillation and fine-tuning. For issue (2), we introduce a feature filtering module that retains only co-visible features, thereby reducing redundancy. To enhance the number of co-visible features, we leverage repeatability loss by formulating the feature filtering process as a keypoint detection task. Furthermore, to minimize hyperparameter tuning, the filtering thresholds are determined automatically. The proposed feature filtering module facilitates more accurate feature interaction while mitigating the impact of irrelevant match candidates. Extensive experiments on multiple datasets demonstrate that our approach significantly reduces computational cost while achieving slightly higher accuracy in pose estimation.