| Description:
|
1 Motivation
A major challenge for emerging IP-based networks is to provide adequate Quality of Experience (QoE) and Quality of Service (QoS) for current and evolving multimedia services and applications such as internet media including over-the-top (OTT) video, and immersive video.
A number of Recommendations have been developed by this Question, in particular:
- In the P.1203 series of standards, an integral model for audiovisual quality-assessment of streaming using reliable transport is described. It enables integral quality estimates for videos between 1 min and 5 min duration, based on short-term audio and video quality modules (Pa/P.1203.2, Pv/P.1203.1), as well as a long-term integration module (Pq/P.1203.3).
- In the P.1204 series of standards, a set of models is described, for bitstream-, pixel-based and hybrid video quality estimation up to 4K resolution, and covering the codecs H.264, HEVC and VP9. It is the first activity of its kind to cover all aforementioned types of relevant video-quality modelling approaches, using an identical dataset for training and validation. Performance figures for the models indicate their strong prediction accuracy.
Both of these standard series can be used for monitoring HTTP adaptive streaming services (such as HLS or DASH), both for TCP or QUIC-type transport. Hence, they represent tools widely applicable in the market.
The previous developments leveraged a unique set of training and validation databases - with a significantly larger number of samples than previously published research - and a modular framework, which enables future, iterative enhancements of the individual modules that are part of the parent recommendations (P.1203 and P.1204). Part of the databases have been published in an open-source manner and have consequently been used in academic research. It will therefore be beneficial for this Question to explore possibilities to further disseminate subjective test database material (video samples, ratings), model training, and/or model reference code in a reproducible manner for the public to validate the results. This in turn should result in a quicker feedback loop in which this Question may obtain comments from different stakeholders and consider those in future updates to Recommendations.
A primary aspect of the continued work is on the inclusion of long-term integration in conjunction with the existing P.1203 and P.1204 standards series. This work has started and is continued here, aiming for a harmonized view on longer-term session quality for the case of adaptive-streaming type services. In particular, the aspect of screen types, different viewing angles and/or display resolutions shall be considered to cover the variety of real-life applications of the models.
Moreover, the inclusion of further video codecs, or different parameterizations of such codecs (e.g., in terms of encoding speed), in updates or extensions of the P.1203 and P.1204 standards will be addressed.
Since today's over-the-top services increasingly involve encrypted transport, passive, network-centric quality monitoring becomes more and more challenging. Bitstream or media-related information may not be readily available, and respective monitoring algorithms may need to apply heuristics to infer application-level performance or quality indicators from network-level features. If network operators wish to assess the quality of the media services offered over their networks, they often need to rely on proprietary solutions. These proprietary offerings are, to the extent known, not using current, standardized approaches, and have not been independently validated. Here, it will be needed to provide the market with means to validate those proprietary tools in terms of their prediction accuracy for different Key Performance Indicators such as buffer behaviour and/or QoE/MOS predictions. Also, the ever-changing nature of OTT services may require the development of a framework to continuously adapt standardized models to the current needs of the market in a more rapid fashion. To address this aspect, the Question will work on approaches for non-intrusive assessment of encrypted video, in collaboration with Q17/12.
Further work items will be addressing extensions of the P.1203 and P.1204 framework towards high dynamic range and wide colour gamut, as well as work on IP-based 360° video quality assessment.
Recent advancements in machine learning/deep learning/artificial intelligence-based approaches in video processing (enhancement, coding, etc.) necessitates the development of video quality models capable of taking into account degradations or possibly improvements introduced by such approaches. To address this, the Question will work towards creation of work items that will build on existing standards such as P.1203 and P.1204 to exploit the modular nature of the model development process to consider machine-learning-related effects in terms of quality prediction.
Some more interactive services have video and audiovisual-quality related components, which could take advantage of the experience and results that have led to the existing P.1200-models developed by this Question. The P.BBQCG work item was started to handle the video quality assessment for cloud gaming services, but also to create a holistic QoE estimation model taking interactivity into account.
Along these lines, the work on extended reality (XR) service QoE will be continued and developed towards a QoE monitoring work item. The exact details of the work will be specified during this Study Period.
The following major deliverables, in force at the time of approval of this Question, fall under its responsibility: P.1200-series.
2 Question
Study items to be considered include, but are not limited to:
- What further aspects of a continued characterization of P.1201, P.1202, P.1203 and P.1204 models should be considered?
- How do P.1201, P.1202, P.1203 and P.1204 need to be maintained, and what further application guidance towards, for example, network-centric monitoring solutions needs to be provided?
- How can existing and future models be adapted/constructed to cover or support an extended scope, e.g., in terms of viewing distance, encoder parameterization, or new error concealment, without having to build a new set of models?
- What are relevant subjective test methodologies, especially when it comes to capabilities of 4K/UHD and 8K, and respective high dynamic range, enhanced colour gamut and high-framerate, and which respective new standards need to be developed (possibly in cooperation with other standardization bodies)?
- How can 8K or HDR video quality be assessed using pixel-, bitstream-based or hybrid modelling approaches?
- How can audiovisual quality be monitored for streams for these cases, and how can audio and video quality be integrated?
- How can bitstream-, signal-based and hybrid models be evaluated for these extended services in a comprehensive standardization activity on the same type of data?
- What relationship exists between the subjective responses of users at the terminals and the objective measurements made from the point at which the assessment system is connected?
- How can audiovisual synchronization be reflected in models such as P.1201, P.1202, P.1203, and P.1204?
- How can long-term integration be addressed for streaming of higher resolutions up to 4 and 8K or HDR content?
- What are the requirements on future updates of the P.1203 and P.1204 standards series for HTTP-based video quality monitoring?
- How can diagnosis assessment be done when using P.1203 and P.1204 standards?
- How can knowledge on short-term measurements and their temporal pooling for longer-term predictions be generalized to complete sessions of multimedia quality monitoring?
- How can video-quality estimation modules for conversational quality estimation models be derived from existing standards or new work within this Question?
- How can video, audiovisual quality and other effects for 360° / omnidirectional video and accompanying audio be monitored?
- How can video- and audiovisual quality prediction best benefit from different machine-learning approaches?
- How can quality of cloud gaming services be assessed?
- How can existing models be adapted/extended to assess the quality of user-generated content?
- How can different XR systems be assessed? This includes defining new test methodologies and instrumental assessment methods.
- How can existing standards (P.1204.X) be extended for quality assessment of AI-related aspects (coding, enhancements, etc.)?
- How can aspects of sustainable streaming, that is, energy or more holistically resource consumption be considered jointly with multimedia quality?
- How can existing models be adapted, or assisted, to be able to handle encrypted streams?
3 Tasks
Tasks include, but are not limited to:
- maintenance of Recommendations P.1201, P.1202, P.1203 and P.1204;
- development of new Recommendation(s) on guidance for the use of P.1201, P.1202, P.1203 and P.1204 in different applications or operational contexts;
- considerations on bitstream-based audio quality evaluation;
- development of tools that are used in the course of model development;
- development of models for assessing video formats such as HDR, wide colour gamut, high framerate;
- development of models for monitoring video quality in the context of conversational and conferencing services;
- development of modelling approaches for quality and QoE evaluation of 360° / omnidirectional video streaming and accompanying audio;
- development of modelling approaches for quality and QoE evaluation of network-based XR systems and services;
- development and maintenance of a new Recommendation on non-intrusive assessment of TLS-encrypted, TCP-based multimedia streaming quality (P.ENATS).
An up-to-date status of work under this Question is contained in the SG12 work programme at https://itu.int/ITU-T/workprog/wp_search.aspx?sp=18&q=14/12.
4 Relationships
Recommendations:
- P.564, G.1000-series, J series recommendations on video quality
Questions:
- 13/12, 17/12
Study groups:
- ITU-T SG13, SG21
- ITU-R WP6C
Other bodies:
- 3GGP SA4, ATIS, Broadband Forum, ETSI TC STQ, HGI, IETF, MPEG, VQEG
WSIS Action Lines:
- C2
Sustainable Development Goals:
- 9
|
