| Work group:
|
Q7/12 (Presentation Web page is available here)
|
| Title:
|
Methodologies, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions
|
| Description:
|
1 Motivation
The work of this Question concerns new methods of assessing the subjective impact of time-varying impairments and includes the design of laboratory and remote testing of speech/noisy-speech/music/mixed content and audiovisual signals. These methods and tools apply to narrowband, wideband, superwideband and fullband audio telephony.
As done so far, considering that the need for standard subjective testing methodologies will continue to exist for the effective assessment of the transmission performance of new communication systems, like speech/music, immersive coders (for audio frequency bandwidths), or other devices and equipment designed for carrying voice and audiovisual signals, the Question will continue to provide the necessary support to produce test/processing plans to execute appropriate subjective tests. Input could also be provided due to the relevant work in other standards organizations, like ISO/MPEG or fora/consortia/partnership projects like 3GPP.
The following major deliverables, in force at the time of approval of this Question, fall under its responsibility: P.85, P.800, P.804, P.805, P.806, P.807, P.808, P.809, P.810, P.811, P.812, P.830, P.835, P.840, P.851, P.852,P.880, P.918, P.1501, P-series Suppls 24, 25, 29, Handbook STP.
2 Question
Study items to be considered include, but are not limited to:
- What new Recommendations need to be developed to evaluate new speech/noisy-speech/music and mixed content quality requirements?
- What new Recommendations need to be developed for multi-dimensional subjective tests in a telephone conversation or multi-party call?
- What enhancements to existing Recommendations need to be defined to improve the evaluation of degradations using immersive codec aspects?
- What enhancements to existing Recommendations need to be defined to improve the subjective evaluation of speech-based or multimodal interactive services?
- What enhanced subjective test methodology is required for evaluating the performance of gaming based applications, in terms of perceived QoS/QoE by game players?
- What new or revised subjective assessment methods are required for evaluating the effects of time-varying impairments (such as delayed packets or packet losses), and what guidance can be provided for the appropriate provision of sample/noise or music material for the testing?
- What modifications to existing or new Recommendations need to be developed to assess new speech/music/mixed content digital coding systems, e.g., narrowband/wideband/superwideband/fullband speech and/or music and/or mixed content and/or immersive codecs operated over fixed and/or 5G mobile networks (including Internet Multimedia Services)?
- What new test plans are needed to evaluate (subjectively) end-to-end communications over fixed and/or 5G mobile networks using data obtained by means of "crowdsourcing"?
- What guidance can be provided for collection and post-screening of subjective test results, and global analysis of results from internationally coordinated exercises in general?
- What are the relationships between various subjective test measures, for example in the auditory modality, between intelligibility, listening effort and QoS/QoE measures?
- What guidance can be provided for collection and evaluation of cultural/language/ nationality dependence of subjective quality?
- What guidance can be provided for collection and evaluation of physiological measures as an additional test method for speech quality assessment?
- Which Questions within SG12, and other standardization activities within ITU, require support for subjective testing?
3 Tasks
Tasks include, but are not limited to:
- maintenance and enhancement of Recommendations in the P-series with regards to subjective testing methods and with regards to the Handbook on Subjective Testing Practical Procedures;
- revise existing Recommendations (e.g., crowdsourcing, gaming, etc.) and draft new ones, e.g., P.PHYSIO and all new Recommendations originated from new work items.
An up-to-date status of work under this Question is contained in the SG12 work programme at https://itu.int/ITU-T/workprog/wp_search.aspx?sp=18&q=7/12.
4 Relationships
Recommendations:
- P-series, G.700-series
Questions:
- 6/12, 9/12, 10/12, 13/12, 15/12, 19/12
Study groups:
- ITU-T SG21
- ITU-R WP5C, WP6C
Other bodies:
- ISO-MPEG, 3GPP, IETF, ETSI, ANSI
WSIS Action Lines:
- C2
Sustainable Development Goals:
- 9
|
| Comment:
|
Continuation of Q7/12
|
|
Rapporteur:
| Mr. | Ludovic | Malfait |
|
|
