(Merging and continuation of Question 14/12 and Question 22/12 studied in 1997-2000)

Type of question

Task-oriented, leading to new and revised ITU-T Recommendations and giving support of new and revised Recommendations which require experimental validation.

Background

The extensive use of digital processing equipment (e.g., echo cancellers, speech codecs, voice activity detectors) in the PSTN, in packetized speech transmission systems, and in wireless communications systems, has produced many benefits for users and operators of the network. In addition to the benefits obtained, such equipment also introduces impairments of various types. The extent to which these impairments have adverse effects on the perceived quality of network connections is of interest. In some cases objective or instrumental methods are available to assist network planners and administrators in assessing the subjective effects of active speech signal processing equipment. However, there are other cases where only subjective evaluations can provide adequate assessment of subjective performance.

In recent years, several issues of subjective performance assessment have arisen, which have been incorporated in Recommendations P.800, P.810, and P.830, P.831, and P.832. However, some issues remain for study. In particular, improved methods of assessing the subjective impact of time-varying impairments and for assessment of speech processing enhancements are required. These methods and tools will be needed for both conventional and wideband telephony.

The Speech Quality Experts Group (SQEG) was created in 1986 and has provided necessary support to plan and execute subjective tests on new speech coders. In particular, SQEG has supported the standardization of new speech coders such as G.722.1, G.728 and G.729 by organizing and interpreting the results of subjective and objective experiments.

Text of the question

considering

a) the need for standard subjective testing methodologies for the effective assessment of the transmission performance of digital systems;

b) standardization of new digital speech coders (telephony-band and higher quality) is likely to continue within ITU for the foreseeable future;

c) there is a need to ensure that the voice quality of such speech coders meets customer expectations;

d) other devices and equipment designed for carrying voice and audiovisual signals will also require evaluation,

the following questions should be studied:

Subjective Assessment Methodologies

1) What methods are required for evaluating the subjective performance of active signal processing devices (especially noise suppression algorithms, voice activity detectors, comfort noise generation)?

2) What new or revised subjective assessment methods are required for evaluating the effects of time-varying impairments such as packet loss, and what guidance can be provided for appropriate duration of speech samples for varying degrees of nonstationarity?

3) What new or revised subjective assessment methods are required for adequate evaluation of music quality in narrowband and wideband telephony?

4) What new or revised subjective assessment methods are required for adequate evaluation of the subjective effects of environmental and other background noises?

5) What guidance can be provided for post-screening of subjective test results?

Subjective Test Plans

6) Which Questions within SG 12, and other standardization activities within ITU, require support for subjective or objective testing?

7) What Recommended methodology is appropriate and what test plan can be elaborated and applied appropriate to the desired level of service quality?

8) When can this testing be carried out, by whom, and what opportunities exist for providing financial support?

9) How should the results of the tests be interpreted, and what conclusions can be drawn?

Objectives and schedule

It is anticipated that new Recommendations on subjective evaluation of packet loss, noise suppression algorithms, and voice activity detectors can be produced in the 2001–2004 Study Period.

Detailed objectives and schedules for test plans and subjective testing support will be generated and updated as part of the on-going work of this Question.

Relationship with other Study Groups

A close relationship will be maintained with ITU-T SGs 15 and 16, and possibly with ITU-T SG 9 and ITU-R SG 8.