Methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions
(Continuation of Question 7/12 - Methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions)Motivation
The work of this Question will be primarily concerned with improved methods of assessing the subjective impact of time-varying impairments and with the design of laboratory testing of speech/noisy-speech/music/mixed content signals, considering the evolution of enhanced telecommunication technologies. These methods and tools will apply to narrowband, wideband, super-wideband and fullband audio telephony.
A Handbook on "Subjective testing practical procedures" was developed during the previous study period for speech and noisy-speech, music and mixed content signals, and will be maintained, adding new procedures whenever they become available and widely used in listening testing laboratories during standard exercises (e.g. for the development of a new coding algorithm within an SDO regular course of action).
As done so far, considering the need for standard subjective testing methodologies will continue to exist for the effective assessment of the transmission performance of new digital systems, like speech/music coders (telephony-band and higher quality bandwidths), or other devices and equipment designed for carrying voice and audiovisual signals, the Question should be mandated to continue to provide the necessary support to produce test/processing plans to execute appropriate subjective tests.
Input could also be provided due to the relevant work in other standards organizations, like ISO/MPEG or fora/consortia like 3GPP and 3GPP2.
The following major Recommendations, in force at the time of approval of this Question, fall under its responsibility:
P.85, P.800, P.805, P.806, P.807, P.810, P.830, P.835, P.840, P.851, P.880, P.1501, P Suppl. 24, P Suppl. 25, Handbook STPQuestion
Study items to be considered include, but are not limited to:
- What new Recommendations need to be developed to evaluate new speech/noisy-speech/music and mixed content quality requirements?
- What new Recommendations need to be developed for a multi-dimensional subjective test methodology suitable to analyse transmitted speech in a telephone conversation?
- What enhancements to existing Recommendations need to be defined to improve the evaluation of degradations due to codec interoperability aspects, e.g. narrowband-to-wideband-to-super-wideband-to-fullband codecs?
- What methods are required for evaluating the subjective performance of active signal processing devices (particularly noise cancellation, generic sound activity detectors / voice activity detectors, comfort noise generation algorithms/devices, bandwidth extension)?
- What enhancements to existing Recommendations need to be defined to improve the subjective evaluation of speech-based or multimodal interactive services?
- What subjective test methodology is required for evaluating the performance of gaming based applications, in terms of perceived quality by game players?
- What new or revised subjective assessment methods are required for evaluating the effects of time-varying impairments (such as packet loss), and what guidance can be provided for the appropriate provision of sample/noise material for varying degrees of nonstationarity?
- What modifications to existing or new Recommendations need to be developed to assess new speech/music/mixed content digital coding systems, e.g. narrowband/wideband/audio speech and/or music and/or mixed content codecs operated over fixed and/or mobile networks (including Internet Multimedia Services)?
- What new test plans are needed to evaluate (subjectively) the above-mentioned new requirements for speech/noise/music quality improvement in end-to-end communications over fixed and/or mobile networks?
- What new test plans are needed to evaluate (subjectively) end-to-end communications over fixed and/or mobile networks using data obtained by means of "crowdsourcing"?
- What guidance can be provided for collection and post-screening of subjective test results, and global analysis of results from internationally coordinated exercises in general?
- What are the relationships between various subjective test measures, for example in the auditory modality, between intelligibility, listening effort and quality measures?
- What guidance can be provided for collection and evaluation of cultural/language/ nationality dependence of subjective quality?
- What guidance can be provided for collection and evaluation of physiological measures as an additional test method for speech quality assessment?
- Which Questions within SG12, and other standardization activities within ITU, require support for subjective testing?
Tasks include, but are not limited to:
- maintenance and enhancement of Recommendations in the P-series with regards to subjective testing methods and with regards to the Handbook on Subjective Testing Practical Procedures;
- draft new and revised Recommendations P.ASPD, P.CROWD, P.GSAD, P.MUS, P.SUSE, P.CLN, P.GAME, P.PHYSIO, P.CQS and maintenance / enhancements to the Handbook on Subjective Testing Procedures.
An up-to-date status of work under this Question is contained in the SG12 work programmehttp://www.itu.int/ITU-T/workprog/wp_search.aspx?q=7/12
- 6/12, 9/12, 10/12, 13/12, 15/12, 18/12, 19/12
- ITU-T SG9, SG16
- ITU-R WP5C, WP6C
- ISO-MPEG, 3GPP, 3GPP2, IETF, ETSI, ANSI