Melanoma is responsible for 60 percent of all malignant skin growths, and the incidence of this kind of cancer has increased dramatically in recent years. It is often detected too late, as diagnostic methods that allow for early detection are laborious and costly. In a full-body skin scan, dermatologists study each pigmented skin lesion in isolation, looking for typical signs of melanoma — a time-consuming and error-prone kind of health check.

The Intelligent Total Body Scanner for Early Detection of Melanoma (iToBoS) EU project has brought together 20 project partners (see list below) with the aim of accelerating and improving the existing examination method with an AI-based full-body scanner. The project has multiple goals: to increase the automated preliminary medical history throughput, optimize early detection of cancers, ease the burden on healthcare professionals, lower the risk to patients, and reduce the number of unnecessary biopsies performed.

The iToBoS scanner developed by project partner Bosch automatically scans the entire body in just six minutes, using a cognitive AI assistant that relies on AI models and XAI methods from Fraunhofer HHI, among other features. Medical professionals see a risk assessment for each individual mole. The scanner is the centerpiece of the cloud-based AI diagnostic platform also developed in the project, which combines health data from various sources, such as patient files, genomic data, and in vivo imaging. The EU is providing 12.1 million euros in funding for the project.

Fast, reliable and highly personalized melanoma diagnosis:

The scanner’s high-resolution cameras are equipped with liquid lenses that mimic the structure of the human eye. Based on two immiscible liquids with different refractive indices, they produce images of unprecedented quality. For highly personalized diagnosis, machine learning is used to integrate the scans together with all available patient data (demographics and information on UV damage, risk group and other factors) into the AI diagnostic platform with the cognitive AI assistant tool.

The high speed at which the scanner operates makes it possible to examine many patients in just a short time, but that is not all. The scans can also be repeated often over a longer period, so they can be compared for effective monitoring of developments in moles and possible skin changes that can be indicative of skin cancer. Multiple specialized AI algorithms combined in the AI assistant are responsible for identifying and tracking moles over time.

XAI methods to verify the methods developed and the data:

The algorithms have to use absolutely reliable and verifiable problem-solving strategies in sensitive areas of application such as medical diagnostics. However, in the past it hasn’t been entirely clear how AI systems make decisions. With their XAI methods used in the project, which include the patented Layer-Wise Relevance Propagation (LRP), Concept Relevance Propagation (CRP) and Prototypical Concept-Based Explanations (PCX) methods, researchers from Fraunhofer HHI are making the AI forecasts explainable, thereby identifying uncertain problem-solving strategies. These methods identify and quantify a broad spectrum of learned decision-making behaviors and identify undesirable decisions even in enormous datasets, prevent the models from making mistakes and ensure that the AI systems used are reliable and safe.

“In the past, AI systems were treated as black boxes. The systems were trusted to do the right things, but unfortunately, that isn’t always the case. With our XAI methods, we’ve succeeded in rendering the solution-finding process of AI systems transparent and overcoming that black box aspect,” says Prof. Wojciech Samek, head of the Artificial Intelligence department at Fraunhofer HHI. “Thanks to our explainable AI methods, we don’t just identify anomalous moles but actually explain at the same time what makes them different. We’re using LRP and CRP to visualize and interpret neural networks and other machine learning models and measure the influence of every input variable in the overall prediction,” adds Dr. Sebastian Lapuschkin, head of the Explainable Artificial Intelligence research group. “Being able to cross-link these huge volumes of data offers the opportunity to learn more about skin cancer and people’s potential for developing it.”

The researchers at Fraunhofer HHI can also use their Reveal to Revise (R2R) method, which was developed as part of the project, to train potential anomalous and erroneous behaviors out of the AI by using the information from their XAI to annotate the data more effectively with the aim of making the entire system more robust and reliable. The ultimate goal of the project is to develop a holistic AI model that uses multimodal datasets to examine patients from a wide variety of perspectives. The meta datasets collected in Queensland, Australia, and in Barcelona as part of the project will also help with this, as they extend the pool of data to other populations from around the world. In the long term, the full-body scanner could also be used to identify other skin diseases such as neurodermatitis.

Project Partners:

BARCO NV, Canfield Scientific, Coronis Computing SL, Fraunhofer HHI, Fundació Clínic per a la Recerca Biomèdica, IBM Israel – Science and Technology Ltd., Institute for Computer Science and Control (SZTAKI), Isahit, Leibniz University Hannover, Melanoma Patient Network Europe, National Technical University of Athens, Optotune AG, RICOH Spain IT Services SLU, Robert Bosch España SA, Torus Actions SAS, Trilateral Research Limited, University of Trieste, University of Queensland, University of Girona (coordinator), V7 Ltd.