This category specifically formulates quantitative testing, evaluation criteria, and benchmark frameworks for various artificial intelligence foundation models (including language, multimodal, vision, and time-series models) and their inference systems. These standards aim to comprehensively measure the performance of foundation models across multiple dimensions such as functionality, accuracy, reliability, and system performance, thereby guiding the industry in scientifically testing and selecting foundation model products.
1.1
General Evaluation & BenchmarkFocuses on the basic evaluation workflows, general criteria, and benchmark testing specifications for foundation models, providing a unified testing yardstick for evaluating multi-dimensional foundational capabilities.
Specifies the overall workflow for assessing foundation models and determines general evaluation criteria covering dimensions such as functionality, accuracy, reliability, security, interactivity, and applicability.
F.748.77 - General framework of foundation models assessment
Defines the benchmark testing framework for foundation models, specifying requirements for testing capabilities (understanding, generation, reasoning), testing datasets (including multiple-choice, subjective questions, and difficulty levels), and testing methods.
1.2 Specific Modality Evaluation
Formulates dedicated evaluation systems and quantitative metrics tailored to the characteristics of specific modalities like vision, time-series, and multimodal, assessing cross-modal semantic association and task execution capabilities.
Establishes a comprehensive testing framework for multimodal foundation models, covering multimodal perception, understanding, and generation capabilities, and provides evaluation methods using metrics like Word Error Rate (WER) and BLEU.
F.748.74 - Framework for the evaluation of multimodal foundation models
Provides evaluation metrics for vision foundation models, covering open-set visual tasks (e.g., object detection, segmentation), visual-language reasoning, and visual embodied intelligence scenarios.
Establishes the evaluation workflow and quantitative metrics for time-series foundation models, assessing the model's performance in forecasting, anomaly detection, classification, and time-series logical reasoning tasks.
1.3 System Adaptation & Inference Performance Focuses on assessing the adaptation and support capabilities of underlying software and hardware systems for foundation models, as well as the actual inference performance of foundation models deployed in cloud or hybrid environments.
ITU-T F.PEM-LLM: Performance evaluation methods for large language model inference service (2025)
Defines performance evaluation methods for large language model inference services, including metrics such as throughput (TPS) and time to first token (TTFT), along with test case designs.
Evaluates the effectiveness of underlying AI hardware/software systems (including CPUs, GPUs, etc.) in adapting to and supporting foundation models, as well as throughput performance under different context lengths.
