Abstract

The use of pre-trained models in medical image classification has gained significant attention due to their ability to handle complex datasets and improve accuracy. However, challenges such as domain-specific customization, interpretability, and computational efficiency remain critical, especially in high-stakes applications such as skin cancer detection. In this paper, we introduce a novel interpretability-assisted fine-tuning framework that leverages category representation to enhance both model accuracy and transparency. Using the widely known HAM10000 dataset, which includes seven imbalanced categories of skin cancer, we demonstrate that our method improves the classification accuracy by 2.6% compared to standard pre-trained models. In addition to precision, we also achieve significant improvements in interpretability, with our category representation framework providing more understandable insights into the model's decision-making process. Key metrics, such as precision and recall, show enhanced performance, particularly for underrepresented skin cancer types such as Melanocytic Nevi (F1 score of 0.94) and Actinic Keratosis (F1 score of 0.66). Furthermore, the prediction accuracy of the proposed model of the top-3 reaches 98. 21%, which is highly significant for medical decision making. This observation in interpretability underscores the value of top-n predictions, especially in challenging cases, to support more informed and accurate decisions. The proposed fusion framework not only enhances predictive accuracy, but also offers an interpretable model output that can assist clinicians in making informed decisions. This makes our approach particularly relevant in medical applications, where both accuracy and transparency are crucial. Our results highlight the potential of fusing pretrained models with category representation techniques to bridge the gap between performance and interpretability in AI-driven healthcare solutions.