|
Work item:
|
H.FDM-AC-V
|
|
Subject/title:
|
Assessment criteria for foundation models: Vision
|
|
Status:
|
[Carried to next study period]
|
|
Approval process:
|
AAP
|
|
Type of work item:
|
Recommendation
|
|
Version:
|
New
|
|
Equivalent number:
|
-
|
|
Timing:
|
-
|
|
Liaison:
|
ISO/IEC JTC1 SC42; ITU-T SG13, 20; ETSI
|
|
Supporting members:
|
China Telecommunications Corporation, Ministry of Industry and Information Technology, China Mobile, State Grid Corporation of China, Zhejiang Lab, China Unicom, Zhejiang Dahua Technology, Alibaba China Co., Ltd.
|
|
Summary:
|
Compared to traditional computer vision models, vision foundation models demonstrate superior generalization capabilities, higher flexibility, and broader adaptability. They are capable of handling challenges of multiple tasks and scenarios, while involving more complex evaluation dimensions, metrics, and datasets. How to assess the performance of vision foundation models, establish evaluation benchmarks, and constructing appropriate test datasets have become hot research topics in academic and industry.
This Recommendation identifies metrics and evaluation methods for vision foundation models. It introduces evaluation object and evaluation methods, specifies requirements and metrics for six critical capabilities of these models: visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, visual object hallucination resistance, and visual embodied intelligence. This draft Recommendation will assist developers in identifying the strengths and limitations of vision foundation models in real-world deployment and application, offering guidance for the design, development, and application.
|
|
Comment:
|
-
|
|
Reference(s):
|
|
|
Historic references:
|
|
Contact(s):
|
|
| ITU-T A.5 justification(s): |
|
|
|
|
First registration in the WP:
2024-06-21 13:57:52
|
|
Last update:
2024-08-20 14:06:51
|
|
