Motivation
The diversification of ICT network solutions, services and technologies causes some problems at the operator's networks, which relate to lack of complete standardized approaches on how to estimate the performance of the vendor's service platforms, e.g. IP Multimedia subsystem (IMS), 5G/IMT 2020.

In general, benchmarking is a common approach used for the measurement and testing of signalling parameters against the performance design objectives, which should help to provide the end to end service delivery and ensure network reliability.
Benchmarking in the case of 5G/IMT-2020 is not limited only to the transport layer and includes performance, quality and reliability of virtual platforms.

The issues of finding the values of network performance and productivity for services with required quality of service (QoS) is an important question for operators and their users.

For example, the Internet-related performance measurement systems, which are publically available on the Internet, do not provide reliable and comparable measurements. Beyond the lack of standardized metrics for Internet speed in ITU-T Recommendations, the obtained test results, which can be achieved by an existing testing method, may vary from the results achieved by other methods. Obviously, the testing results depend on the network segments which are used during e2e connection. Particularly, it is not possible to guarantee that the e2e connection is based on the telecom operator’s network only and does not include other network segments which may belong to other operators.

The development of a unified approach to Internet-related performance measurement is important for all ICT stakeholders.
Also, this standardized approach will support the development of the “Framework for ICT service delivery with the guaranteed QoS and requested bitrate on fixed & mobile data networks, for development of efficient economic mechanisms and models of interaction in the "operator-provider-user" chain”.

In addition, one of the Pillars of the ITU Conformity and interoperability (C&I) programme is aimed at assisting the creation of regional testing centres. A testing centre may be implemented using cloud computing technologies that have the ability to enable a new concept of remote testing which may be named Testing as a Service (TAAS). This new ITU T research area will allow testing laboratories to identify the requirements and principles of the remote testing procedures.

The characteristics of 'adaptive networks' such as virtualization, self-organization, self-configuration, self-optimization, self-healing and self-learning offer huge advantages in future networks. While technologies such as network functions virtualization (NFV), self-organizing networks (SON), mobile edge computing (MEC) and autonomic network infrastructure (AFI) may not each exhibit all the characteristics, they do have one thing in common: they are all dynamic rather than static, reacting to dynamic traffic conditions, applications, service demands as well as to changes in the eco-system environment. The task is to develop a methodology (guide) that would extend current experiences and testing approaches for 5G/IMT-2020.

NOTE − The network performance QoS/QoE, including the networks and services KPIs, parameters and requirements, are defined by ITU- T SG12 and related SDOs.

Question
Study items to be considered include, but are not limited to:

• What kind of equipment could be benchmark tested?
• What kind of test cases could be used for benchmark testing?
• What type of traffic could be simulated for benchmark testing?
• What design objectives need to be benchmark tested?
• How standardized network parameters should be tested?
• What test suites for standardized network parameters, including those defined in the framework of Internet related performance measurements, need to be developed?

NOTE – ITU-T Q.3960 (Framework of Internet-related performance measurement) was approved on 6 July 2016;

• How to test the standardized network parameters, such as e2e bit rate within the fixed and mobile operator’s network?
• How to test standardized network parameters, such as e2e bit rate in Internet (that is between users of the operator’s network and a particular Internet resource)?
• What procedures need to be developed to implement remote testing?
• What is the network architecture to be used for remote testing?
• Who can be involved in the remote testing and their roles

Tasks
Tasks include, but are not limited to:

• determine the types of equipment, which could be benchmark tested;
• develop the test cases for benchmark testing;
• identify the type of traffic to be simulated for benchmark testing;
• define the design objectives that need to be benchmark tested;
• determine the test cases for standardized network parameters
• develop test suites for standardized network parameters, including those defined in the framework of Internet-related performance measurements;
• specify how to test the standardized network parameters, such as e2e bit rate, within the fixed and mobile operator’s networks;
• specify how to test the standardized network parameters, such as e2e bit rate beyond the fixed and mobile operator’s network (that is between users of the operator’s network and a particular Internet resource);
• develop the procedures needed to implement remote testing;
• specify the network architecture to be used for remote testing;
• research on roles and responsibilities of stakeholders who may be involved in the remote testing;
• develop a methodology (guide) that would extend current experiences and testing approaches to 5G/IMT-2020;

​An up-to-date status of work under this Question is contained in the SG 11 work programme
(http://itu.int/ITU-T/workprog/wp_search.aspx?Q=9/11)

Relationships

Recommendations

• Q, Y, H, I, M, F, P, E and G-series

​Questions

• 11/11, 12/11, 14/11 and 10/11

​Study groups

• ITU‑T SG3 on policy issues
• ITU‑T SG12 on parameters and requirements of QoS
• ITU‑T SG13 on future networks architecture and Cloud computing
• ITU‑T SG16 on multimedia services and applications
• ITU-D SG2 on remote test centres

​Standardization bodies

• ETSI
• IETF