Metrics reloaded: recommendations for image analysis validation - Université de Rennes Access content directly
Journal Articles Nature Methods Year : 2024

Metrics reloaded: recommendations for image analysis validation

Lena Maier-Hein (1) , Annika Reinke (1) , Patrick Godau (1) , Minu Tizabi (1) , Florian Buettner (1) , Evangelia Christodoulou (1) , Ben Glocker (2) , Fabian Isensee (1) , Jens Kleesiek (3) , Michal Kozubek (4) , Mauricio Reyes (5) , Michael Riegler (6) , Manuel Wiesenfarth (1) , A Emre Kavur (1) , Carole Sudre (7) , Michael Baumgartner (1) , Matthias Eisenmann (1) , Doreen Heckmann-Nötzel (1) , Tim Rädsch (1) , Laura Acion (8) , Michela Antonelli (7) , Tal Arbel (9) , Spyridon Bakas (10) , Arriel Benis (11) , Matthew Blaschko (12) , M Jorge Cardoso (13) , Veronika Cheplygina (14) , Beth Cimini (15) , Gary Collins (16) , Keyvan Farahani (17) , Luciana Ferrer (18) , Adrian Galdran (19) , Bram van Ginneken (20) , Robert Haase (21) , Daniel Hashimoto (10) , Michael Hoffman (22) , Merel Huisman (20) , Pierre Jannin (23, 24) , Charles Kahn (10) , Dagmar Kainmueller (25) , Bernhard Kainz (26) , Alexandros Karargyris (27) , Alan Karthikesalingam (28) , Florian Kofler (29) , Annette Kopp-Schneider (1) , Anna Kreshuk (30) , Tahsin Kurc (31) , Bennett Landman (32) , Geert Litjens (20) , Amin Madani (33) , Klaus Maier-Hein (1) , Anne Martel (22) , Peter Mattson (34) , Erik Meijering (35) , Bjoern Menze (36) , Karel Moons (37) , Henning Müller (38) , Brennan Nichyporuk (39) , Felix Nickel (40) , Jens Petersen (1) , Nasir Rajpoot (41) , Nicola Rieke , Julio Saez-Rodriguez (42) , Clara Sánchez (43) , Shravya Shetty (44) , Maarten van Smeden (45) , Ronald Summers (17) , Abdel Taha (46) , Aleksei Tiulpin (47) , Sotirios Tsaftaris (48) , Ben van Calster (49) , Gaël Varoquaux (50) , Paul Jäger (1)
1 DKFZ - German Cancer Research Center - Deutsches Krebsforschungszentrum [Heidelberg]
2 Imperial College London
3 Universität Duisburg-Essen = University of Duisburg-Essen [Essen]
4 MUNI - Masaryk University [Brno]
5 UNIBE - Universität Bern / University of Bern
6 UiT - The Arctic University of Norway [Tromsø, Norway]
7 UCL - University College of London [London]
8 CONICET - Consejo Nacional de Investigaciones Científicas y Técnicas [Buenos Aires]
9 McGill University = Université McGill [Montréal, Canada]
10 University of Pennsylvania
11 HIT - Holon Institut of Technology
12 KU Leuven - Catholic University of Leuven = Katholieke Universiteit Leuven
13 King‘s College London
14 ITU - IT University of Copenhagen
15 BROAD INSTITUTE - Broad Institute of MIT and Harvard
16 University of Oxford
17 NCI-NIH - National Cancer Institute [Bethesda]
18 CONICET-UNC Córdoba Argentina
19 UPF - Universitat Pompeu Fabra [Barcelona]
20 Radboud University Medical Center [Nijmegen]
21 TU Dresden - Technische Universität Dresden = Dresden University of Technology
22 University of Toronto
23 LTSI - Laboratoire Traitement du Signal et de l'Image
24 Centre Hospitalier Universitaire de Rennes [CHU Rennes] = Rennes University Hospital [Ponchaillou]
25 University of Potsdam = Universität Potsdam
26 FAU - Friedrich-Alexander Universität Erlangen-Nürnberg = University of Erlangen-Nuremberg
27 IHU Strasbourg - L'Institut hospitalo-universitaire de Strasbourg
28 DeepMind [London]
29 Helmholtz AI
30 EMBL - European Molecular Biology Laboratory
31 SBU - Stony Brook University [SUNY]
32 Vanderbilt University [Nashville]
33 University Health Network [Toronto, ON, Canada]
34 Google Inc [Mountain View]
35 University of New South Wales [Kensington]
36 UZH - Universität Zürich [Zürich] = University of Zurich
37 Universiteit Utrecht / Utrecht University [Utrecht]
38 UNIGE - Université de Genève = University of Geneva
39 Mila - Institut québécois d’intelligence artificielle
40 UKE - Universitaetsklinikum Hamburg-Eppendorf = University Medical Center Hamburg-Eppendorf [Hamburg]
41 University of Warwick [Coventry]
42 Universität Heidelberg [Heidelberg] = Heidelberg University
43 UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam
44 Google Inc.
45 UMCU - University Medical Center [Utrecht]
46 TU Wien - Fakultät für Mathematik und Geoinformation [Wien]
47 University of Oulu
48 Edin. - University of Edinburgh
49 LUMC - Leiden University Medical Center
50 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
Ben Glocker
Michal Kozubek
Spyridon Bakas
Arriel Benis
Veronika Cheplygina
Beth Cimini
Gary Collins
Daniel Hashimoto
Michael Hoffman
Charles Kahn
Florian Kofler
  • Function : Author
Anna Kreshuk
Bennett Landman
Anne Martel
Peter Mattson
Nasir Rajpoot
Nicola Rieke
Shravya Shetty
  • Function : Author
Ronald Summers
Aleksei Tiulpin

Abstract

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Fichier principal
Vignette du fichier
2206.01653.pdf (35.18 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04477840 , version 1 (04-04-2024)

Identifiers

Cite

Lena Maier-Hein, Annika Reinke, Patrick Godau, Minu Tizabi, Florian Buettner, et al.. Metrics reloaded: recommendations for image analysis validation. Nature Methods, 2024, 21 (2), pp.195-212. ⟨10.1038/s41592-023-02151-z⟩. ⟨hal-04477840⟩
42 View
3 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More