Page 34 - Shaping ethics, regulation and standardization in AI for health

Shaping ethics, regulation and standardization in AI for health

TG-Ophthalmo, in collaboration with the FGAI4H ML Audit activity, has completed tasks
such as drafting an Audit Verification Checklist and starting the setup of benchmarking for
Diabetic Retinopathy (DR) on the ML Audit platform. Completed practice tasks include platform
registration, updating challenge configuration files, creating and hosting a text prediction
challenge, participating in it, and creating annotation and submission files. Remaining tasks
include Dockerized model testing with images and obtaining an undisclosed test data set.

A�4�9 DEL 10�10: FG-AI4H Topic Description Document for the Topic group
on Outbreak detection (TG-Outbreaks)

Summary: This TDD specifies a standardized benchmarking for AI in outbreak detection (TG-
Outbreaks). Its primary purpose is to specify a standardized benchmarking framework for artificial
intelligence algorithms used in public health for detecting disease outbreaks. The document
covers various essential aspects, including the definition of the AI task, ethical and regulatory
considerations, existing benchmarking work, and the proposed benchmarking methodology
of the topic group, aiming to create a basis for evaluating and comparing AI solutions in this
critical area of public health.

The AI task involves planning and implementing data collection for health events, environmental
contamination, weather, and watershed ecological data in eThekwini, South Africa. This effort,
led by Woodco and the University of KwaZulu-Natal, aims to address the high incidence of
diarrhoeal disease in marginalized communities. Key aspects include community engagement,
ethical considerations, and data privacy. The collected data, including health case counts,
waterborne pathogen testing, and satellite data, will be used to predict and prevent diarrhoeal
disease outbreaks through an algorithm, ultimately improving sanitation and health outcomes
in areas with limited infrastructure.

Data simulation was still ongoing during final submission of this TDD. Furthermore, approval to
share simulated data was also not yet legally cleared. Thus, these results could not be included
in this deliverable.

A�4�10 DEL 10�12: FG-AI4H Topic Description Document for the Topic group
on AI for radiology (TG-Radiology)

Summary: This TDD outlines the work of the Topic Group on AI for radiology (TG-Radiology),
addressing the global shortage of radiologists by exploring AI solutions. A significant challenge
identified is the lack of standardized methods for benchmarking and evaluating AI radiological
systems, particularly in ensuring they can generalize across diverse data and handle complex
cases. The document proposes a radiograph-agnostic platform and framework for standardized
benchmarking, emphasizing a "Precision Evaluation" approach that assesses AI performance
across various demographic groups and geographical locations. Ethical considerations,
including bias and data privacy, are also discussed as crucial aspects of deploying AI in radiology.

A�4�11 DEL 10�14: FG-AI4H Topic Description Document for the Topic group
on Symptom assessment (TG-Symptom)

Summary: This document describes the work towards the specification of a standardized
benchmarking for AI-based symptom assessment systems. In recent years, one promising
approach to meet the challenging shortage of doctors has been the introduction of AI-based

24