|
Work item:
|
F.FDM-AC-BK
|
|
Subject/title:
|
Assessment criteria for foundation models: Benchmark
|
|
Status:
|
[Carried to next study period]
|
|
Approval process:
|
AAP
|
|
Type of work item:
|
Recommendation
|
|
Version:
|
New
|
|
Equivalent number:
|
-
|
|
Timing:
|
-
|
|
Liaison:
|
-
|
|
Supporting members:
|
-
|
|
Summary:
|
Foundation models are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As foundation models continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Benchmark testing of foundation models, which evaluates the performance of foundation models by designing reasonable testing tasks and evaluation datasets, is currently the most recognizable way from industry to academia. As a result, by formulating the criteria and standardizing the framework of benchmark, which includes evaluation metrics, datasets, methods, and tools, can help model developers and users to obtain high-quality evaluation results. This draft recommendation provides requirements of foundation models and identifies evaluation methods for foundation models. It can help industry participants analyze and optimize foundation models, as well as help users build and select products that meet the requirements.
|
|
Comment:
|
-
|
|
Reference(s):
|
|
|
Historic references:
|
|
Contact(s):
|
|
| ITU-T A.5 justification(s): |
|
|
|
|
First registration in the WP:
2024-06-19 12:02:53
|
|
Last update:
2024-09-20 10:48:09
|
