1 Type of Question

Task-oriented, leading to new ITU-T Recommendations.

2 Background

In digital systems the quality of each communication medium is influenced by a number of interacting factors, such as source coding and compression, bit rate (fixed or variable), delay, bandwidth, synchronization between the media, transmission impairments, and many others. New services that use IP, wireless, mobile, NGN, ISDN, B-ISDN, ATM, etc. are providing ubiquitous access for multimedia services. Conversational audiovisual multimedia includes such applications as videoconferencing, personal computer desktop conferencing and videotelephony. In order to develop the two-way measurement techniques required for conversational applications a basis in one-way audio and video quality evaluation must first be defined and validated.

Objective methods

Current objective quality measuring techniques do not predict user opinion on the perceived audiovisual quality with desirable accuracy. It is therefore desirable to identify objective techniques for measuring the various individual and combined effects of factors such as digital compression, transmission, storage, and others on the perceived quality of audiovisual systems. It is also important to verify that these techniques are meaningful by correlating proposed objective tests with corresponding subjective test data.

Subjective methods

There is a need to continue to develop new subjective methods to address new audiovisual services. The perceived quality depends on the kind of application and on the tasks the applications are used for. For example, in a free conversation through a videophone, the perceived quality may primarily depend on delay, lip-synchronization and audio quality, while in a mainly one-way application like remote-teaching the perceived quality could be primarily related to the quality of graph and low motion picture sequences.

The work on subjective assessment resulted in several new Recommendations in this area.

- P.910, Subjective Video Quality Assessment Methods for Multimedia Applications

- P.911, Subjective Audiovisual Quality Assessment Methods for Multimedia Applications

- P.920, Interactive Test Method for Audiovisual Communications

- P.930, Reference Impairment System for Video.

The work on objective methods yielded two Recommendations:

- P.931, Multimedia Communication Delay, Synchronization, and Frame Rate Measurements

- P.861, Objective Quality Measurement of Telephone-Band (300-3 400 Hz) Speech Codecs.

These Recommendations and those of other study groups in both Sectors will be utilized, where possible, in the work of this Question.

Much of the work on this Question (and its predecessors) was and will be done in conjunction with the Video Quality Experts Group (VQEG).

3 Text of the Question

1) Interaction of media: What subjective and objective measurement methods should be used to evaluate end-to-end quality of each medium (e.g. video, audio) and the interactions between the media, with particular attention to the audiovisual quality assessment of systems used for videoconferencing/videotelephony and other conversational multimedia services? What are the quality levels that can be defined by objective or subjective methods in different applications (or tasks) taking into account the interactions between media?

2) Transmission errors: What objective methods could be used for in-service measurement and monitoring of transmission systems for multimedia services in the presence of transmission errors? What new subjective measurement methods should be used for the evaluation of transmission quality of real time audiovisual services by expert observers resulting in the identification of specific flaws in the transmission equipment or environment? What procedures should be used, and which dimensions, transforms, and partial or differential signals should be viewed by experts to evaluate specific impairments of real time audiovisual services?

What objective and subjective methods can be used to evaluate audiovisual signals with time-varying quality?

3) Impairment characterizations: Among the most significant factors (e.g. spatial resolution, temporal resolution, colour fidelity, audio and visual artefacts, media synchronization, delay, etc.) affecting the overall quality of multimedia services, what objective and subjective methods assess the extent of or can differentiate between these factors? How can the mutual interaction between these factors be objectively and subjectively measured with respect to their influence on overall audiovisual quality? For what applications can the assessment methods be shown to be useful and robust over a range of conditions?

4) Evaluation of specific services: What assessment methods (objective and subjective) can be used to characterize the quality effects of Multipoint Conference Units for audiovisual communication and other new audiovisual services such as remote monitoring and mobile audiovisual communication?

5) Test methodologies: What objective or subjective methods and assessment tools are required to fully describe perceived audiovisual impairments in terms of measurable system parameters? What kind of references should be used in subjective tests?

6) Combination of test results: In some cases it may be useful to combine objective measures (e.g. video measures, audio measures, media synchronization) to provide a single figure of merit. In this regard, which objective measures and/or techniques should be combined, and in what manner, so that the figure of merit correlates satisfactorily with subjective test results?

7) Test sequences: While the library of test sequences has increased greatly during the last study period, there is still a need for more test sequences, especially those with audio included. Which audiovisual test material (e.g. audiovisual test sequences) can be used for subjective and objective evaluations? In addition to the definitions of SI and TI in P.910, which criteria (objective and/or subjective) should be used to characterize and classify audiovisual test material?

8) Validation and applicability of objective methods: There are three basic methodologies of objective picture quality measurement. Full-reference (FR) uses the full bandwidth video input. Reduced-reference (RR) uses lower bandwidth features extracted from the video input. No-reference (NR) has no information about the video input. What objective methodology should be used for different multimedia applications? What subjective methods should be used to validate each of the three basic objective methodologies?

4 Objectives and schedule

Subjective quality evaluation in multimedia services requires on the one hand the continuous updating of Recommendations P.910, P.920 and P.930 and also the definition of new task-oriented/application-dependent evaluation methods for the combined evaluation of audio and video signals. A new Recommendation utilizing expert viewers is expected in 2002.

Objective methods are still young in their development but progress is being made rapidly. The first draft of a Recommendation covering the framework for the objective assessment of video quality (P.ovq-frame) is expected to be available in 2001. The first draft of a Recommendation on the objective assessment of video quality (P.OVQ) is expected in 2002. A first draft of a Recommendation on objective assessment of audiovisual quality (P.OAV) is expected in 2003.

5 Relationship with other study groups

This work will most likely be of interest to ITU-T Study Groups 2, 12, 13, 14,15 and 16 and ITU-R Study Groups 6 and 8.