Results from the
Experts Meeting on Subjective and Objective Video Quality Assessment
Turin, Italy
October 14-16, 1997

The status of this meeting was changed from a meeting of the Joint Rapporteurs’ Group to a meeting of experts on subjective and objective video quality assessment. This was done to ensure that the representatives from the ITU-R WP11E were given equal status to the members from the ITU-T.
The first two days were devoted to objective measures of video quality, while the third day was devoted to subjective test methods.  

Objective measures
The goal of the meeting was to define a test plan for validating objective measures of video quality to be included in ITU recommendations.
Topics of discussion included:
Purpose of the test.
	One or more objective video quality assessment methods are sought to meet a number of measurement needs.
	See Table 1 below for a depiction of some of these needs.
	It was decided that the first effort to test objective measures for video quality would concentrate on the TV3 class and include smaller examples of MM4, TV2, and perhaps TV1.

Criteria of Acceptance
There was no agreement on a criteria of acceptance.  A number of data analysis techniques were proposed including:
	Correlation coefficient, Rank-order correlation, RMS Error, weighted RMS Error, and Variance-Covariance Analysis.
Other considerations were discussed including monotonicity, clustering, robustness, graceful degradation, non-linear regression methods, outlier considerations, generalizability, color error sensitivity, and complexity of measurement method.
This topic was referred to an ad hoc group for further work.

Fairness of Test
To avoid proponents utilizing apriori knowledge, it was decided to require proponents to submit an executable program (complete with any input parameters) to an Independent Lab by June 1, 1998.  Furthermore, all test scenes and the final HRC list will be decided by the Independent Lab and Selection Committee.

Test Plan for Subjective Test
The test will include approximately 200 clips chosen (intelligently) from a set of 20 HRCs and 20 test scenes.  The test will be of the DSCQE type from Rec. 500. The test scenes will probably be 8 seconds, but 5 second test scenes are being considered.
This topic was referred to an ad hoc group for further work.. A Subjective test plan will be completed by January 1, 1998 and submitted to Study Groups 9, 12, and WP11E for comments.

Test Plan for Objective Test
Items to be decided include data analysis techniques, criteria for acceptance, computer platform on which the submitted executable code will run.
This topic was referred to an ad hoc group for further work. An Objective test plan will be completed by January 1, 1998 and submitted to Study Groups 9, 12, and WP11E for comments.

Video Classes Table and Definitions 
Some changes were made to the Video Classes and Attributes tables (See Tables 2 and 3). The definitions of TV2 and TV3 were changed to include examples from Cable Television. The Video Quality Classes table was revised by deleting the two columns concerning Latency/Delay Variation and Channel Impairments. It was felt that more work is needed in these two areas to provide accurate and relevant data. This is still considered a working document to be included in future Draft New Recommendations.

Delay measurements

The current draft of P.DEL was presented and comments were solicited.

Established Ad-hoc Groups:

Subjective Test Plan
Co-chairs:	Philip Corriveau, CRC, Canada
Alexander Schertz, IRT, Germany

Objective Test Plan (including acceptance criteria)
Co-chairs:	Mihir Ravel, Tektronix, United States
John Beerends, KPN, the Netherlands

Video Classes Table and Definitions 
Co-chairs:	David Fibush, Tektronix, United States
Arthur Webster, NTIA, United States

Independent Lab and Selection Committee
Co-chairs: 	Laura Contin, CSELT, Italy
Philip Corriveau, CRC, Canada

Subjective Testing Labs:
CRC (Canada), RAI (Italy), IRT(Germany) (Subjective. or Objective), CCETT (France). 
Possibly Swiss Telecom, CSELT(Italy), and one Lab from Japan

Objective Testing Labs
IRT (Germany) (Subjective. or Objective), CRC (Canada), CSELT (Italy), Possibly one from Japan

Subjective test methods
The third day meeting was devoted to methodologies for subjective quality evaluation. This  meeting was useful for the exchange of experience among the experts, that also gave hints for possible developments in different ITU’s Study Groups (SGs). Common activities of different ITU SGs were not proposed for the time being. For this reason  the following sections will not include any decision and will not illustrate any workplan. 
ITU activities related to subjective video quality evaluation
The ITU activities related to subjective quality evaluation were illustrated by Mr. Flowers (ITU-T SG9), Mr. Schertz (ITU-R SG11) and Ms. Contin (ITU-T SG12)
Mr.  Flowers illustrated the activities developed in the framework of Question 22/9 (doc.19), with particular attention to Recommendation J.SVQ1 (doc.20), that is under development in SG9. This recommendation should include subjective test methods to evaluate picture quality in a home viewing environment and consider different kind of links. The test method proposed is the SSCQE.
Mr. Schertz reported some studies carried out in the framework of ITU-R WP11E on the contextual effects. The main outcome of this work is that, at the bitrates taken into account, the DSCQS methods is more stable than the other two test methods investigated (i.e. DSIS and comparison). 
Mr. Schertz also presented a work carried out at IRT about home viewing conditions. This triggered a discussion about the problem of setting ‘typical’ home viewing conditions. It came out that it depends on the Country. In North America it seems to be between 7.1 and 8 H, independently on the size of the monitor, while in Japan big monitors are preferred, also when the viewing distance cannot be changed (due to room size limitation).
Ms. Contin illustrated the content of the three recommendations developed by SG12 in the area of video quality assessment, that are: P.910, P.920 and P.930. The first one addresses one-way video quality subjective test methods, similar to those used for evaluating television picture quality, the second one specifies test protocols for conversation tests, to be used for interactive applications, while the third one specifies the principles to develop a video impairment reference system.
SG12 has planned to revise the above mentioned recommendations, but the main effort is now addressed to the development of a new recommendation on one-way audiovisual quality assessment.
New subjective video quality assessment methods
Mr. Corriveau illustrated the SSCQE method, recently included in recommendation BT.500. As this is a quite new methodology, some studies are still necessary to improve it. For example it would be useful to have a method to calculate a unique figure of merit on the whole sequence.
Ms. Pascal presented a work carried out by CNET in collaboration with CCETT, where the SSCQE was used in an interactive test (doc. 16). The results show that this method can be used also to test interactive systems, like those for video conference, although it is not yet clear whether it provides results as accurated as those achievable with methods described in recommendation P.920.
 
Ms Contin illustrated the DSCQE (Double Stimulus Continuous Quality Evaluation) method, that was proposed in MPEG-4 for testing error robustness of video coding schemes. It is based on the continuous evaluation, exactly as SSCQE, but in the DSCQE the viewers must evaluate the impairment of the test sequence with respect to a suitable reference. To this aim both the reference and the test conditions are displayed at the same time. 
It was stressed that for some application and for some coding scheme fidelity may be more important than quality. In this case scales different from the currently used impairment scale could be more appropriate. Mr. Lubin and Mr. Ravel proposed to use scales that exclude emotional response. For example impairment perceptability 
scale (imperceptible, slightly perceptible, mildly perceptible, clearly perceptible, extremely perceptible) could replace the used impairment annoyance scale (imperceptible, perceptible but not annoying, slightly annoying, annoying, very annoying).

 (Audio)visual libraries
ITU-R has developed a library of video sequences, that are available also to ITU-T for study purposes. Ms. Contin will try to provide both ITU-T and ITU-R with  video and audiovisual sequences used by MPEG.
Criticality of sequences
Two contributions were presented on this issue (doc. 10 and 11). They will be taken into account in the selection of test sequences to be used in the validation of the objective methods.
Audiovisual test methods
Three contributions related  to one-way audiovisual quality evaluation(doc. 8, 9 and 21) were presented by Paul Bowman and Arthur Webster. Moreover J. Bereends briefly illustrated an experiment carried out at KPN.  In general these studies were aimed at investigating the interaction of audio and video quality and their impact on the overall audiovisual quality. It was observed that the results obtained are not in disagreement either among them or with previous studies carried out in Bellcore.
Laura Contin drove the attention on the need of a recommendation on subjective audiovisual test methods. Viewing and listening conditions were discussed for these tests. It was observed that the laboratory set up depends on the goal of the test. To evaluate the performance of systems under real using conditions, PC monitors may be used, provided that they are correctly calibrated. 
Viewers
The use of either experts or not-experts in subjective tests was discussed. It was emphasized the importance of the training, because the evaluation may change as the viewers become used with the kind of impairments considered in the test. 
It was also discussed how long should be the participation of each subject. The conclusion was that each session shouldn’t last more than 30 minutes and overall the viewing time required to each subject should not exceed 1 hour.

Conclusions

Based on the commitments taken during the Turin meeting, the activities aimed at preparing the verification tests for objective video quality measurements will be continued by correspondence in the relevant ad hoc groups. 
Progress reports of each ad hoc group will be sent to the reflector set up by Arthur Webster.

List of Attendees
John Beerends, KPN, the Netherlands
Paul Bowman, BT, Great Britain
Noel Chateau, France Telecom/CNET, France
Stephanie Colonnese, FUB, Italy
Laura Contin, CSELT, Italy
Philip Corriveau, CRC, Canada
David Fibush, Tektronix, United States
Brian Flowers, EBU/BBC
Takahiro Hamada, KDD, Japan
Jeff Lubin, Sarnoff, United States
Dominique Pascal, France Telecom/CNET, France
Stephane Pefferkorn, CCETT, France
Mihir Ravel, Tektronix, United States
Alexander Schertz, IRT, Germany
Massimo Visca, RAI, Italy
Arthur Webster, NTIA, United States
Stefan Winkler, EPFL, Switzerland

Revised list of HRCs.
Note: This list is general.  Specific hardware or software implementations as well as final resolution, profile, level, and/or
GOP structure will be left to the Independent Selection Committee.
	Bit rate	Res	Method	Comments
1	768 kb/s	CIF	H.263
2	1.5 Mb/s	CIF	H.263
3	2   Mb/s	3/4	mp@ml	This is a horizontal resolution reduction only
4	2   Mb/s	3/4	sp@ml		
5	3   Mb/s		mp@ml
6	4.5 Mb/s		mp@ml
7	4.5 Mb/s. 		mp@ml	Composite NTSC and/or PAL
8	4.5 Mb/s	3/4	sp@ml	At least one commercial encoder does this
9	6   Mb/s		mp@ml		
10	8   Mb/s		mp@ml
11	8   Mb/s 		mp@ml	Composite NTSC and/or PAL
12	8 & 4.5 Mb/s		mp@ml	Two codecs concatenated
13	12  Mb/s		mp@ml
14	18  Mb/s		422p@ml	IB only, true SX encoder preferred
15	21  Mb/s		422p@ml	Large GOP, new EBU standard
16	36  Mb/s		422p@ml	I only
17	36  Mb/s		422p@ml	IB only	Eight generations, spatial and GOP shifts
18	n/a		n/a	VHS (must be fully time base corrected)
19	n/a		n/a	Multi-generation Betacam (4 or 5, composite or component)
20	4.5 Mb/s		mp@ml 	w/errors TBD (to be determined)
21	36  Mb/s			422p@ml	I only,  w/errors TBD (perhaps a lower bit rate
For 20 and 21, artifacts are to be kept within the same quality range as the other impairments in the test. 

List of Documents
jrg002.rtf - 2nd version of Table defining video quality classes
jrg003.rtf - Sarnoff JND Vision Model-Algorithm Description and Testing (Presentation by Jeffrey Lubin of Sarnoff to T1A1.5 Aug.97)
jrg004.pdf - Survey of Image and Video Quality Metrics based on Vision Models (Presentation by Christian van den Branden Lambrecht of Hewlett Packard to T1A1.5 Aug. 97)
jrg005.rtf - Draft P.DEL, ATT
jrg006.doc - Proposals on Methods for Determining Algorithms for Recommendation Agendum #3), ATT
jrg007.doc - Proposals on Test Material Production and Other Issues (Agendum #4 and #5), ATT
jrg008.pdf  - Preliminary Results, Subjective Desktop Video Teleconferencing Audiovisual Test, NTIA/ITS
jrg009.pdf - Single-Ended Multimedia Subjective Test NTIA/ITS
jrg010.pdf - Subjective and Objective Measures of  Scene Criticality, NTIA/ITS
jrg011.rtf - Objective Picture Quality Evaluation for the Digital Broadcasting by Introduction of Criticality, NHK Science & Technical Research Laboratories
jrg012.lis - Comments for the Turin ITU Experts Meeting on Subjective and Objective Audio/Visual Quality Assessment Methods, CableLabs
jrg014.rtf - IRT/Tektronix Investigation of Subjective and Objective Picture Quality for 2-10 Mbit/sec MPEG-2 Video: Phase 1 Results, IRT and Tektronix
jrg015.rtf - Video Codec Evaluation Scheme and Implementation Based on Characteristics of Human Visual Perception, KDD
jrg016.doc - Some Results from a methodological preliminary experiment for assessing the audio-visual quality in group communication, France Telecom/CNET
jrg017.rtf - Objective Picture Quality Scale for Digital Compressed Picture for Broadcast, NHK
jrg018 - Objective Video Quality Assessment by Image Comparison, TDF-CR2 France, et.al.
jrg019 - ITU-T SG9 Question 22, Measurement and Control of the Quality of Service of Digital Television Transmission
Networks
jrg020 - Com-9-15, Draft New ITU-T Recommendation J.SVQ1- Subjective Picture Quality Assessment for Digital Cable 
	Television Systems, Ass. Rapporteur A. Karwowska-Lamparka
jrg021 - Objective Performance Assessment: Video Quality as an Influence on Audio Perception, BT
jrg022 - List of participants
jrg023 – Excerpt from Quovadis document:  Transmission errors
jrg024 - Excerpt from Quovadis document:  Conclusions
jrg025 - Flow diagram for validation testing
jrg026 - Revised HRC list
jrg027 - Statistical Analysis of Image-Fidelity Ratings, Sarnoff (Mike Brill)
jrg028 - ITU-R WP11E Revised a question on Objective Quality

Flow diagram for Objective Measurements Test

TABLE 1 Video Quality Classes (vertical) and Test Scope Levels

	OUT OF SERVICE TESTS	IN SERVICE TESTS	
CLASS	1	2	3	4	5		1	2	3	4	5	
TV1												
TV2												
TV3												
MM4												
MM5												
MM6												

from Document #6 for Levels 1, 2, 3, & 4, Al Morton, ATT

1. to assess the continued operational readiness of a video transmission system.
2. to compare the relative levels of impairment between two different systems at 
	the same or different bit rates (both may exhibit supra-threshold impairment).
3. to assess the level of sub-detection threshold impairment of a system, in such a way 
	that the assessments of individual systems can be combined in calculations
	to design a perceptually loss-less system.
4. to assess the level of supra-threshold impairment of a system, in such a way that the 
	assessments of individual systems can be combined in calculations to design a system 
	to a specific perceived impairment level.
5. to replace subjective tests.

Table 2 Definitions of Classes
TV 0
	Loss-less:	
TV 1	Contribution:  Used for complete post production, many edits and processing layers, intra-plant transmission. Also used for remote site to plant transmission.	
TV 2	Primary distribution: Used for simple modifications, few edits, character/logo overlays, program insertion, and inter-facility transmission. A broadcast example would be network to affiliate transmission. An example in a cable system would be a regional downlink facility to a local headend.
	
TV 3
	Secondary distribution: Used for delivery to home/consumer (no changes). An example in a cable system would be from a local headend to a home.
	
MM 4
	All frames encoded.  Low Artifacts. Usually   30 fps.	
MM 5
	Frames May be Dropped at Encoder.  Perceptual artifacts possible, but useful for designed tasks.	
MM 6
	Series of Stills. Not Intended to provide full motion. (Examples: Surveillance, Graphics)	

Table 3  Attributes of Classes
Class of Operation	Spatial Format	Typical Coding Algorithms1	Delivered Frame Rate	Nominal Bit Rate, Mbit/s	
TV 0	Rec. 601	Loss-less	Max FR	270 or 360 	
TV 1	Rec. 601	MPEG-2	Max FR	18 to 50 	
TV 2	Rec. 601	MPEG-2,
DV	Max FR	10 to 25	
TV 3	Rec. 601	MPEG-2	Max FR
Infrequent
(occasional)
frame repetition	1.5 to 10	
MM 4	Rec. 601	H.262,
MPEG-2	(15-) 30 fps	1.5	
MM 4	CIF	H.263	~30 fps	0.768	
MM 5	CIF		10-30 fps		
MM 5	QCIF		1-15 fps		
MM 5	Sub-QCIF	H.263	>0 fps
<30 fps		
MM 6	16CIF	H.263			

Note 1:  Proprietary coding algorithms are also possible.  If used, they should provide performance at least as good as the specified standard algorithm.