1 Scope
1.1 Limitations
2 References
3 Definitions
3.1 Terms defined elsewhere
3.2 Terms defined in this Recommendation
4 Abbreviations and acronyms
5 Conventions
6 Source stimuli
6.1 Source signal recordings
6.2 Video considerations
6.3 Audio considerations
6.4 Audiovisual considerations
6.5 Duration of stimuli
6.6 Number of source stimuli
7 Test methods, rating scales and allowed changes
7.1 List of methods
7.2 Acceptable changes to the methods
7.3 Discouraged but acceptable changes to the methods
8 Environment
8.1 Controlled environment
8.2 Public environment
8.3 Viewing distance
9 Subjects
9.1 Number of subjects
9.2 Subject population
9.3 Sampling subjects
9.4 Sampling techniques
10 Experimental design
10.1 Size of the experiment and subject fatigue
10.2 Special considerations for transmission error, rebuffering and
audiovisual synchronization impairments
10.3 Special considerations for longer stalling events
10.4 Pre-pilot testing and pilot testing
10.5 Study design
11 Experiment implementation
11.1 Informed consent
11.2 Overview of subject screening
11.3 Optional pre-screening of subjects
11.4 Post-screening of subjects
11.5 Instructions and training
11.6 Study duration, sessions and breaks
11.7 Stimuli play mechanism
11.8 Voting
11.9 Questionnaire or interview
12 Data analysis
12.1 Documenting the experiment
12.2 Calculate MOS or DMOS
12.3 Evaluating objective metrics
12.4 Significance testing, subject bias and standard deviation of
scores
12.5 Ratings from multiple laboratories
13 Elements of subjective test reporting
13.1 Documenting the test design
13.2 Documenting the subjective testing
13.3 Data analysis
13.4 Additional information
Annex A – Method for post-experimental screening of subjects using Pearson
linear correlation
A.1 Screen by PVS
A.2 Screen by PVS and HRC
Appendix I – Sample informed consent form
Appendix II – Sample instructions
Bibliography