Skin Lesion Classification using Dermoscopic Images and Clinical Metadata: Insights from Multimodal Models

Ahammed, Sakib, Cui, Xia ORCID: https://orcid.org/0000-0002-1726-3814, Lu, Wenqi and Yap, Moi Hoon ORCID: https://orcid.org/0000-0001-7681-4287 (2025) Skin Lesion Classification using Dermoscopic Images and Clinical Metadata: Insights from Multimodal Models. In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 222-230. Presented at IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 11 June 2025 - 12 June 2025, Nashville, TN, USA.

Preview

Accepted Version Available under License Creative Commons Attribution. Download (3MB) | Preview

Abstract

State-of-the-art methods primarily use dermoscopic skin images to classify lesions. Incorporating clinical metadata to improve melanoma diagnosis has increased popularity in the community. This study explores whether a single modality (dermoscopic images) or multimodality (dermoscopic images and clinical metadata) enhances performance for skin lesion classification. In particular, it tries to answer the question by comparing three multimodal architectures. This paper introduces the ISIC-DICM-17K dataset, a curated balanced dataset comprising 17,060 dermoscopic images and clinical metadata with an equal distribution of melanoma and non-melanoma classes. We evaluate the impact of including clinical metadata on multimodal model performance using both supervised and transfer learning models. Our results show that including metadata significantly enhances supervised learning models' performance, while all transfer learning models outperform supervised learning models with 3 % to 12 % performance improvement. Our statistical and visual analyses highlight the importance of clinical metadata in improving feature clustering and class separability, and the HeatMap index analysis shows the effectiveness in identifying relevant features on lesion images. Our GitHub repository (accessible at https://github.com/mmu-dermatology-research/isic-dicm-17k) contains the ISIC-DICM-17K dataset and image IDs referencing the original ISIC dataset sources.