Webinar

Datasets through the Lđź‘€king-Glass

Datasets through the Lđź‘€king-Glass is a webinar series focusing on the data aspects on learning-based methods. Our aim is to build a community of scientists interested in understanding how the data we use affects the algorithms and society as a whole, instead of only optimizing for a performance metric. We draw inspiration from a variety of topics, such as data curation to build datasets, meta-data, shortcuts, fairness, ethics and philosophy in AI.

All previous talks where the authors have agreed to share the talk, can be found in our YouTube playlist.

Next webinar: TBA

Date: TBA

Where: Zoom:

Speakers:

  • Leo Joskowicz - School of Computer Science and Engineering at the Hebrew University of Jerusalem, Israel

Title: Observer variability in the AI machine learning zoo

Abstract: Deep learning models have become the method of choice for the automatic detection and segmentation of structures in medical images. Supervised learning methods require training datasets with radiologists’ annotations. Supervised learning methods yield excellent results but rely on large, annotated datasets to train them, which is expensive and time-consuming, as it requires expert annotators. A variety of methods, including few-shot learning, semi-supervised learning and active learning have been proposed. However, none of them yields satisfactory results, especially for small structures such as lesions and tumors. Moreover, observer variability is seldom reported. This talk discusses the key role that the production of manual annotations and their cost has in developing RadApps and how it can be optimized.

Short bio: Leo Joskowicz is a Professor at the School of Computer Science and Engineering at the Hebrew University of Jerusalem, Israel since 1995. He is the founder and director of the Computer-Aided Surgery and Medical Image Processing Laboratory (CASMIP Lab). Prof. Joskowicz is a Fellow of the IEEE (Institute of Electrical and Electronics Engineers), ASME (Association of Mechanical Engineers), and MICCAI (Medical Image Processing and Computer Aided Intervention) Societies. He is the past President of the MICCAI Society, was the Secretary General of CAOS International (International Society of Computer Aided Orthopaedic Surgery) and ISCAS (International Society for Computer Assisted Surgery). He is the recipient of the 2010 Maurice E. Muller Award for Excellence in Computer Assisted Surgery by CAOS International and the 2007 Kaye Innovation Award. He has published two books and over 350 technical works including book chapters, conference and journal papers and editorials and has 15 issued patents. He is on the Editorial Boards of several journals, including Medical Image Analysis , IEEE Trans. Medical Imaging , Int. J. of Computer Aided Surgery , and Computer Aided Surgery and has served on numerous related program committees. He is the co-founder of HighRAD, an oncology AI imaging company with FDA-cleared products.

  • Laura Quinn - University of Birmingham, UK

Title: Interobserver Variability in Breast Screening: A Comparison of Measures

Abstract: Interobserver variability between readers is an important consideration in diagnostic imaging, with direct implications for patient care. Using data from a breast cancer screening programme, this talk will compare commonly used variability measures and demonstrate how different approaches can lead to different conclusions about agreement between readers. The session will discuss the strengths and limitations of these measures, together with practical considerations for selecting, interpreting, and reporting interobserver variability in diagnostic imaging research.

Short bio: Laura Quinn is a Research Fellow in Medical Statistics at the University of Birmingham. Her research focuses on diagnostic test evaluation, particularly interobserver variability in diagnostic imaging, which was the focus of her PhD. She also conducts research on methods for evaluating healthcare performance over time and is currently establishing a research programme on the evaluation and monitoring of AI in diagnostic imaging.

  • Kumar Abhishek - Simon Fraser University’s Medical Image Analysis Lab, Canada

Title: Styles, Signals, and Scale: Learning from Annotator Disagreement in Skin Lesion Segmentation

Abstract: Manual segmentations of skin lesions are routinely treated as ground truth. In practice, however, different annotators often produce noticeably different lesion boundaries, even when delineating the same image. These differences arise from variations in expertise, annotation tools, and individual interpretation. Rather than viewing this disagreement as noise or collapsing it into a single “ground truth”, this talk argues for treating it as data. I will introduce IMA++, the largest public multi-annotator skin lesion segmentation dataset, curated from the ISIC Archive and containing 14,967 dermoscopic images and 17,684 segmentation masks from 16 annotators. In addition to the masks themselves, the dataset includes metadata about the annotation tool used and the level of expertise of the manual reviewer. Unlike most multi-annotator datasets, IMA++ has an incomplete image–annotator graph, reflecting the practical constraints of real annotation workflows where not every annotator labels every image. Building on this data, I will present two lines of work. The first introduces StyleSeg, which discovers plausible, diverse, and semantically consistent segmentation “styles” from a corpus of image-mask pairs, without requiring information about which annotator produced each mask. The second examines inter-annotator agreement and shows that malignant lesions systematically, and statistically significantly, exhibit higher disagreement than benign ones. It also demonstrates that agreement can be predicted from the image itself, and that incorporating this prediction as an auxiliary learning task improves diagnostic performance.

Short bio: Kumar Abhishek is a PhD candidate at Simon Fraser University’s Medical Image Analysis Lab, supervised by Prof. Ghassan Hamarneh. He holds an MSc from SFU and a B.Tech. from IIT Guwahati. His research develops machine learning methods for skin image analysis, with a recent focus on data: annotator variability and segmentation styles, dataset curation, and data quality audits of popular dermatological benchmarks. His work has appeared at conferences such as CVPR, MICCAI, ISBI, and MIDL, and in journals including Medical Image Analysis, Scientific Reports, and Scientific Data, and has received four Best Paper Awards: MICCAI ISIC Skin Image Analysis Workshop 2025 and 2024, IEEE ISBI 2025, and CVPR ISIC Skin Image Analysis Workshop 2021.

Previous talks:

All previous abstracts can be found here.

  • S01E01 - Dr. Roxana Daneshjou (Stanford University School of Medicine, Stanford, CA, USA). 27th Feb 2023. Challenges with equipoise and fairness in AI/ML datasets in dermatology
  • S01E02 - Dr. David Wen (Oxford University Clinical Academic Graduate School, University of Oxford, Oxford, UK). 27th Feb 2023. Characteristics of open access skin cancer image datasets: implications for equitable digital health
  • S01E03 - Prof. Colin Fleming (Ninewells Hospital, Dundee, UK). 27th Feb 2023. Characteristics of skin lesions datasets
  • S02E01 - Prof. Amber Simpson (Queen’s University, Canada). 5th June 2023. The medical segmentation decathlon
  • S02E02 - Dr. Esther E. Bron (Erasmus MC - University Medical Center Rotterdam, the Netherlands). 5th June 2023. Image analysis and machine learning competitions in dementia
  • S02E03 - Dr. Ujjwal Baid (University of Pennsylvania, USA). 5th June 2023. Brain tumor segmentation challenge 2023
  • S03E01 - Dr. Thijs Kooi (Lunit, South Korea). 18th September 2023. Optimizing annotation cost for AI based medical image analysis
  • S03E02 - Dr. Andre Pacheco (Federal University of EspĂ­rito Santo, Brazil). 18th September 2023. PAD-UFES-20: the challenges and opportunities in creating a skin lesion dataset
  • S04E01 - Dr. Jessica Schrouff (Google DeepMind, UK). 4th December 2023. Detecting shortcut learning for fair medical AI
  • S04E02 - Rhys Compton and Lily Zhang (New York University, USA). 4th December 2023. When more is less: Incorporating additional datasets can hurt performance by introducing spurious correlations
  • S04E03 - Dr. Enzo Ferrante (CONICET, Argentina). 4th December 2023. Building and auditing a large-scale x-ray segmentation dataset with automatic annotations: Navigating fairness without ground-truth
  • S05E01 - Hubert Dariusz ZajÄ…c and Natalia-Rozalia Avlona (University of Copenhagen, Denmark). 25th March 2024. Ground Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI
  • S05E02 - Dr. Annika Reinke (DKFZ, Germany). 25th March 2024. Why your Dataset Matters: Choosing the Right Metrics for Biomedical Image Analysis
  • S05E03 - Alceu Bissoto and Dr. Sandra Avila (UNICAMP, Brazil). 25th March 2024. The Performance of Transferability Metrics does not Translate to Medical Tasks
  • S06E01 - Hava Chaptoukaev and Maria Zuluaga (EURECOM, France). 24th February 2025. Acquiring, curating and releasing a multi-modal dataset for stress detection: ambitions, achievements, mistakes and lessons learned
  • S06E02 - Alice Jin (Massachusetts Institute of Technology, USA). 24th February 2025. Fair Multimodal Checklists for Interpretable Clinical Time Series Prediction
  • S06E03 - Malih Alikhani and Resmi Ramachandranpillai (Northeastern University, USA). 24th February 2025. Towards Equity: Overcoming Fairness Challenges in Multimodal Learning
  • S07E01 - Amelia JimĂ©nez-Sánchez (IT University of Copenhagen, Denmark). 12th May 2025. In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review
  • S07E02 - Tiarna Lee (King’s College London, UK). 12th May 2025. Racial bias in cardiac imaging
  • S07E03 - Dewinda J. Rumala (UNIVERSA AI, Switzerland). 12th May 2025. Seeing the Same Brain Twice: Data Leakage and Identity Bias in Brain MRI Analysis
  • S08E01 - Mamunur Rahaman (University of New South Wales, Australia). 20th October 2025. Advancing Computational Pathology: Multimodal Datasets and Deep Learning Insights
  • S08E02 - David Restrepo (CentraleSupĂ©lec, UniversitĂ© Paris-Saclay, France). 20th October 2025. Opening Eyes: Advancing Equitable AI Through Open Ophthalmology Data
  • S08E03 - Yuki Arase (School of Computing, Institute of Science Tokyo, Japan). 20th October 2025. Japanese Medical Text Simplification Using Patient Blogs
  • S09E01 - Evangelia Christodoulou (German Cancer Research Center (DKFZ), Germany). 2nd March 2026. From Metrics to Meaning: Making Evaluation Matter in Medical Imaging AI
  • S09E02 - Abhishek Singh Sambyal (University of Oulu, Finland). 2nd March 2026. Beyond Accuracy: Understanding Calibration in Medical Image Classification
  • S09E03 - Raghavendra Selvan (University of Copenhagen, Denmark). 2nd March 2026. Carbon footprint of Medical Image Analysis and Mitigation Strategies

All previous abstracts can be found here.

Organizers

Amelia Jiménez-Sánchez at the Universitat de Barcelona (Spain), Théo Sourget & Veronika Cheplygina at the IT University of Copenhagen (Denmark), and Steff Groefsema at the University of Groningen (the Netherlands). This project has received funding from the Independent Research Fund Denmark - Inge Lehmann number 1134-00017B.

Newsletter

If you want to receive information about upcoming seminars, please sign up to our mailing list. We pick the GDPR-compliant Brevo (formerly Sendinblue) as our mail provider. If you have any concerns relating to our data handling, please read our privacy notice.

Please be aware that many mail providers are tagged as junk, and the confirmation email might end up in your spam folder. Double check if your confirmation email is there. The sender will be PURRlab @ IT University of Copenhagen (amji @ itu.dk). Please add this sender to your contacts. If you have any problems subscribing to our mailing list, please contact Amelia.