Page 28 - Shaping ethics, regulation and standardization in AI for health

Shaping ethics, regulation and standardization in AI for health

undisclosed test data set – per use case of each topic area – that will not be made accessible
to the AI developers.

A�3�5 DEL 5�3: Data annotation specification

Summary: Data annotation would be one of the most dependable factors on model performance,
it serves as one important aspect of data quality control on artificial intelligence for health (AI4H).
ITU-T FG-AI4H Deliverable DEL5.3 gives a general guideline of data annotation specification,
including definition, background and goals, framework, standard operating procedure, scenario
classifications and corresponding criteria, as well as recommended metadata and so on. A
questionnaire is attached to seek input and collaboration with topic groups regarding data
annotation.

A�3�6 DEL 5�4: Training and test data specification

Summary: ITU-T FG-AI4H Deliverable DEL5.4 provides guidelines on the systematic way of
preparing technical requirements specifications for datasets used in the training and testing of
machine learning models, and it discusses the best practices of data quality assurance aimed
at minimizing the data error risks during the training and test data preparation phase of the
machine learning process lifecycle.

A�3�7 DEL 5�5: Data handling

Summary: ITU-T FG-AI4H Deliverable DEL5.5 outlines how data will be handled, once accepted.
Health data is one of the most valuable and sensitive types of data. Handling this kind of data
is often associated with a strict and factual framework defined by data protection laws. It is
important to set a strict data policy which will ensure confidence in FG-AI4H, not only between
contributors but across all stakeholders. There are two major issues that the data handling
policy should address: a) compliance with regulations dealing with the use of personal health
data; and b) non-disclosure of the undisclosed test data held by FG-AI4H for the purpose of
model evaluation.

A�3�8 DEL 6: AI training best practices specification

Summary: Machine learning models for artificial intelligence (AI) in health are deployed in high-
impact tasks. As a result, it is important to follow best practices for training and documentation
to achieve maximum performance and transparency. The first part of this document provides a
review of best practices for proper AI model training. The second part of this document provides
guidelines for model reporting.

A�3�9 DEL 7: Artificial intelligence for health evaluation considerations

Summary: In this document, considerations on the evaluation and benchmarking of health AI
are presented, novel characteristics of health AI validation and evaluation are identified, and
the concept of standardized model benchmarking is discussed. Moreover, requirements for
a benchmarking platform are considered in detail, and best practices for the health AI model
assessment are collected from selected sources.

18