Recommendation ITU-T F.748.26 (02/2024) Technical specification for artificial intelligence cloud platforms: Performance evaluation
Summary
History
FOREWORD
Table of Contents
1 Scope
2 References
3 Definitions
     3.1 Terms defined elsewhere
     3.2 Terms defined in this Recommendation
4 Abbreviations and acronyms
5 Conventions
6 Overview of AI cloud platform performance evaluation framework
     6.1 Evaluation object
     6.2 Evaluation principle
     6.3 Workflow of performance evaluation framework
7 Configuration specification
     7.1 Computing resource cluster configuration
     7.2 Node configuration
     7.3 Software configuration
     7.4 Physical environment configuration
8 Evaluation workloads and metrics
     8.1 Operation level
          8.1.1 Computing operations
               8.1.1.1 Workloads for computing operations
               8.1.1.2 Metrics for computing operations
          8.1.2 Network operations
               8.1.2.1 Workloads for network operations
               8.1.2.2 Metrics for network operations
          8.1.3 IO operations
               8.1.3.1 Workloads for IO operations
               8.1.3.2 Metrics for IO operations
     8.2 Model level
          8.2.1 Workloads for model level benchmark
          8.2.2 Metrics for model training task
          8.2.3 Metrics for algorithm development task
          8.2.4 Metrics for model inference task
          8.2.5 Metrics for model deployment task
     8.3 Platform level
          8.3.1 Platform level task information
          8.3.2 Metrics for platform level performance
9 Requirements on evaluation results
     9.1 Benchmark report
     9.2 Benchmark materials
Appendix I  Evaluation suggestions
     I.1 Evaluation program
     I.2 Dataset preparation
     I.3 Evaluation workloads
Bibliography