|
Work item:
|
F.PEM-LLM
|
|
Subject/title:
|
Performance evaluation methods for large language model inference service
|
|
Status:
|
Under study
|
|
Approval process:
|
AAP
|
|
Type of work item:
|
Recommendation
|
|
Version:
|
New
|
|
Equivalent number:
|
-
|
|
Timing:
|
2027-12 (Medium priority)
|
|
Liaison:
|
ISO/IEC JTC1 SC42, ITU-T SG13
|
|
Supporting members:
|
CAICT, Zhejiang Lab, Alibaba, State Grid Cooperation of China, China Telecom
|
|
Summary:
|
Large language models are rapidly integrating into multimedia applications such as audio and video, driving a significant improvement in cross-modal understanding and generation capabilities. This trend places higher demands on the performance of model inference services. Through a standardized performance evaluation framework, this Recommendation systematically defines testing specifications for large language model inference services, including performance testing workflow, dataset construction principles, measurement methods for core performance indicators, and result analysis paradigms. These efforts aim to ensure the comparability and credibility of test results, thereby providing a reference for technology selection during the development and use of AI-enabled multimedia applications.
|
|
Comment:
|
-
|
|
Reference(s):
|
|
|
Historic references:
|
|
Contact(s):
|
|
| ITU-T A.5 justification(s): |
|
|
|
|
First registration in the WP:
2025-11-17 11:29:44
|
|
Last update:
2025-11-25 17:13:50
|
