Development of models and tools for multimedia quality assessment of packet-based video services
(Continuation of Question 14/12 - Development of parametric models and tools for multimedia quality assessment)Motivation
A major challenge for emerging IP-based networks is to provide adequate Quality of Experience (QoE) and Quality of Service (QoS) for new multimedia services and applications such as IPTV, internet media including over-the-top (OTT) video, and immersive video. In previous study periods, new Recommendations have been developed by Q.14/12:
- P.1201 ("P.NAMS") and P.1202 ("P.NBAMS") have successfully been developed to meet the challenge of assessing the quality for video transmitted over UDP in a scalable manner, proposing different no-reference, trace analysis based audio, video, and audiovisual quality models.
- An extension of P.1201 was created (Appendix III, "P.NAMS-PD") addressing progressive download video quality monitoring, hence considering TCP-based transmission and respective re-buffering-related stalling.
Since TCP based and OTT video services are very popular, methods like P.1201 and P.1202 were addressed to include TCP and OTT video, considering both adaptive and non-adaptive streaming.
This is reflected in the standardization activity P.NATS Phase 1 (the ITU-T P.1203 series of standards) on non-intrusive assessment of TCP-based streaming. The P.NATS Phase 1 development activity, targeting integral quality estimates for videos between 1min and 5min duration, was started in Study Period 2013-2016, and finalization is planned for 2016, focusing on the integral model including the short-term audio and video quality modules (Pa, Pv), as well as the long-term integration module (Pq).
P.NATS Phase 2 will focus on improved and extended short-term video quality modules Pv, including aspects such as 4k/UHD and additional codecs such as HEVC and VP9. The topic of 4k/UHD and the related increased spatial resolution and colour gamut are topics of increasing relevance for the market. Hence, upcoming standards on monitoring solutions should include such new technology. For P.NATS Phase 2, the Question collaborates with the AVHD project of the Video Quality Experts Group (VQEG). In this project, various types of video quality models will be validated on the same dataset: Signal-based (Full Reference, No Reference, Reduced Reference), bitstream-based (Modes 0, 1, 2 and 3 according to the P.NATS Phase 1 work) and hybrid models.
Since today’s over-the-top services increasingly involve encrypted transport, mid-network quality monitoring becomes more and more challenging. Bitstream or media-related information may not be readily available, and respective monitoring algorithms may need to apply heuristics. If network operators wish to assess the quality of the media services offered over their networks, they often need to rely on proprietary solutions that are not using current, standardized approaches. Here, it will be needed to provide the market with means to validate certain proprietary tools in terms of their predictions of Key Performance Indicators such as buffer behaviour and/or MOS predictions. To address this aspect, the Question will work on the newly created work item P.ENATS (Encrypted non-intrusive assessment of TCP-based streaming), in collaboration with Q17/12.
Further work items may be addressing extensions of the P.NATS framework towards High Dynamic Range and Wide Color Gamut, as well as work on IP-based 360° video quality assessment.
The following major Recommendations, in force at the time of approval of this Question, fall under its responsibility:
Study items to be considered include, but are not limited to:
- What further aspects of a continued characterization of P.1201 and P.1202 models should be considered?
- How do P.1201 and P.1202 need to be maintained, and what further application guidance towards, for example, network-centric monitoring solutions needs to be provided?
- What are relevant subjective test methodologies, especially when it comes to capabilities of 4k/UHD and respective high dynamic range and enhanced colour gamut, and which respective new standards need to be developed (possibly in cooperation with other standardization bodies)?
- How can 4k/UHD video quality be assessed using bitstream-based monitoring approaches?
- How can audiovisual quality be monitored for streams with encryption such as transport layer security (TLS)?
- How can bitstream-, signal-based and hybrid models be evaluated in a comprehensive standardization activity on the same type of data?
- What relationship exists between the subjective responses of users at the terminals and the objective measurements made from the point at which the non-intrusive assessment system is connected?
- How can audiovisual synchronization be reflected in parametric models such as P.1201, P.1202 and P.1203?
- What are the requirements on future updates of P.NATS for TCP- and HTTP-based video quality monitoring?
- How can knowledge on short-term measurements and their temporal pooling for longer-term predictions be generalized to complete sessions of multimedia quality monitoring?
- How can different quality-related aspects of a multimedia service be combined to an integral quality index for general service monitoring? An example is that of an IPTV session with content selection, viewing, zapping, EPG usage, etc.
Tasks include, but are not limited to:
- maintenance of Recommendation P.1201 Parametric non-intrusive assessment of audiovisual media streaming quality;
- maintenance of Recommendation P.1202 Parametric non-intrusive bitstream assessment of video streaming quality;
- development of new Recommendation(s) on guidance for the use of P.1201 and P.1202 in operational contexts;
- considerations on bitstream-based audio quality evaluation;
- development of tools that are used in the course of model development;
- maintenance of Recommendation P.1203 Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport and its extensions to further media formats;
- development and maintenance of a new Recommendation on non-intrusive assessment of TLS-encrypted, TCP-based multimedia streaming quality (P.ENATS);
- development of a new Recommendation for temporal pooling of multimedia quality for longer sequences (P.MMSTP – multi-media session temporal pooling);
- development of a new Recommendation on integral index of quality for general service monitoring (P.INQX).
An up-to-date status of work under this Question is contained in the SG12 work programmehttp://www.itu.int/ITU-T/workprog/wp_search.aspx?q=14/12
- P.564, G.1000-series, J series recommendations on video quality
- ITU-T SG13, SG16
- ITU-R WP6C
- ETSI TC STQ, IETF, ATIS, VQEG, HGI, Broadband Forum