Motivation

The work of this Question will be primarily concerned with improved methods of assessing the subjective impact of time-varying impairments and with the design of laboratory testing of speech/audio/music and audiovisual quality interactions. These methods and tools apply for conventional (narrowband), wideband, superwideband, fullband / audio telephony. A Handbook of subjective testing practical procedures was started to be developed during the previous Study Periods, limited to speech and noisy-speech testing, and will be complemented with the procedures for testing of audio/music signals and audiovisual quality interactions.

Considering the need for standard subjective testing methodologies will continue to exist for the effective assessment of the transmission performance of new digital systems, like speech/music/audiovisual codecs (telephony-band and higher bandwidth quality), or other devices and equipment designed for carrying voice and audiovisual signals, the Question should be mandated to provide the necessary support to produce test/processing plans to execute appropriate subjective tests, including measurements of intelligibility.

Considering the need to investigate the cultural/language/nationality dependence of subjective quality, this project will cover not only speech quality, but also video and multimedia quality.

This Question will cover the subjective quality aspects of the overall service related to the performance of speech and gesture recognition devices, speech and visual output, and interaction management.

Input could also be provided due to the relevant work in other standards organisations, like ISO/MPEG or fora/consortia like 3GPP and 3GPP2.

The following major Recommendations/Supplements, in force at the time of approval of this Question, fall under its responsibility:

P.85, P.800, P.805, P.810, P.830, P.835, P.840, P.851, P.880, Suppl. 24 to P-Series Recommendations.

Question

Study items to be considered include, but are not limited to:

• What new Recommendations need to be developed to evaluate new speech/music (stereo / multichannel) quality requirements?
• What enhancements to existing Recommendations need to be defined to improve the evaluation of degradations due to codec interoperability aspects, e.g. narrowband-to-wideband / superwideband / fullband codecs ?
• What enhancements to existing Recommendations need to be defined to improve the evaluation of encoded speech + environmental noise, or both kinds of signals (separately)?
• What modifications to existing or new Recommendations need to be developed to assess new speech/music digital coding systems, e.g. narrowband / wideband / superwideband / fullband / audio speech and/or music codecs operated over fixed and/or mobile networks (including Internet multimedia services)?
• What methods are required for evaluating the subjective performance of active signal processing devices (especially noise cancellation, voice activity detectors, and comfort noise generation algorithms/devices)?
• What new or revised subjective assessment methods are required for evaluating the effects of time-varying impairments such as packet loss, and what guidance can be provided for the appropriate provision of sample material for varying degrees of non-stationarity?
• What new test plans are needed to evaluate (subjectively) the above mentioned new requirements for speech/noise/music/audiovisual quality improvement in end-to-end communications over fixed and/or mobile networks?
• What guidance can be provided for collection and post-screening of subjective test results?
• Which quality aspects are important for the users of speech-based or multimodal interactive services? How can these aspects be quantified with subjective evaluation methods?
• How are the subjective quality aspects of the overall service related to the performance of the underlying technological devices, like speech and gesture recognition, speech and visual output, and interaction management? Is it possible to predict service quality on the basis of measurable parameters?
• Which Questions within SG 12, and other standardization activities within ITU, require support for subjective testing?
• Considerations on how to help measure and mitigate climate change.

Tasks include, but are not limited to:

• Maintenance and enhancement of Recommendations in the P-series with regards to subjective testing methods and of the Handbook on Subjective Testing Procedures.
• Draft new and revised Recommendations P.805, P.ASPD, P.MUS, P.SAD and complete a Handbook on Subjective Testing Procedures can be produced in the 2009–2012 Study Period:

An up-to-date status of work under this Question is contained in the SG 12 Work Programme http://www.itu.int/ITU-T/workprog/wp_search.aspx?isn_sp=545&isn_sg=551

Relationships

Recommendations: P-series, G.700 Series

Questions: 6/12, 8/12, 9/12, 15/12

Study Groups: ITU-T SG 9, SG 15, SG 16 and ITU-R SG 5

Standardization bodies: ISO-MPEG, 3GPP, 3GPP2, IETF, ETSI