e-space
Manchester Metropolitan University's Research Repository

    Analysis of the ISIC image datasets: Usage, benchmarks and recommendations

    Cassidy, B ORCID logoORCID: https://orcid.org/0000-0003-3741-8120, Kendrick, C, Brodzicki, A, Jaworek-Korjakowska, J and Yap, MH ORCID logoORCID: https://orcid.org/0000-0001-7681-4287 (2022) Analysis of the ISIC image datasets: Usage, benchmarks and recommendations. Medical Image Analysis, 75. 102305. ISSN 1361-8415

    [img]
    Preview
    Published Version
    Available under License Creative Commons Attribution Non-commercial No Derivatives.

    Download (4MB) | Preview

    Abstract

    The International Skin Imaging Collaboration (ISIC) datasets have become a leading repository for researchers in machine learning for medical image analysis, especially in the field of skin cancer detection and malignancy assessment. They contain tens of thousands of dermoscopic photographs together with gold-standard lesion diagnosis metadata. The associated yearly challenges have resulted in major contributions to the field, with papers reporting measures well in excess of human experts. Skin cancers can be divided into two major groups - melanoma and non-melanoma. Although less prevalent, melanoma is considered to be more serious as it can quickly spread to other organs if not treated at an early stage. In this paper, we summarise the usage of the ISIC dataset images and present an analysis of yearly releases over a period of 2016 - 2020. Our analysis found a significant number of duplicate images, both within and between the datasets. Additionally, we also noted duplicates spread across testing and training sets. Due to these irregularities, we propose a duplicate removal strategy and recommend a curated dataset for researchers to use when working on ISIC datasets. Given that ISIC 2020 focused on melanoma classification, we conduct experiments to provide benchmark results on the ISIC 2020 test set, with additional analysis on the smaller ISIC 2017 test set. Testing was completed following the application of our duplicate removal strategy and an additional data balancing step. As a result of removing 14,310 duplicate images from the training set, our benchmark results show good levels of melanoma prediction with an AUC of 0.80 for the best performing model. As our aim was not to maximise network performance, we did not include additional steps in our experiments. Finally, we provide recommendations for future research by highlighting irregularities that may present research challenges. A list of image files with reference to the original ISIC dataset sources for the recommended curated training set will be shared on our GitHub repository (available at www.github.com/mmu-dermatology-research/isic_duplicate_removal_strategy).

    Impact and Reach

    Statistics

    Activity Overview
    6 month trend
    225Downloads
    6 month trend
    123Hits

    Additional statistics for this dataset are available via IRStats2.

    Altmetric

    Repository staff only

    Edit record Edit record