1
Scope
2
References
3
Definitions
4
Abbreviations and acronyms
5
Conventions
6
Subjective test and objective algorithms
6.1
Aspects related to subjective testing
6.2
Aspects related to objective algorithms
7
Evaluation framework
7.1
Data preparation
7.2
Analysis types
7.3
Prediction on a numerical quality scale
7.4
Uncertainty of subjective results
7.5
Statistical evaluation metrics
7.6
Statistical significance evaluation
7.7
Statistical evaluation in the context of subjective uncertainty: epsilon
insensitive rmse and its statistical significance
7.8
Statistical evaluation of the overall performance
8
Guidance on algorithm selection
8.1
Per experiment performance
8.2
Overall figure of merit
8.3
Worst performance cases
8.4
Averaging statistical metrics across experiments
9
Special cases
9.1
Evaluation of algorithms with more than one output
9.2
Evaluation of algorithms against pre-defined minimum performance
requirements
10
Demonstration cases
Appendix I – Algorithm mapping to the subjective scale
Appendix II – The impact of the third order versus first order
mapping
II.1
Application of third order and first order mappings
II.2
Gain of third order mapping
Appendix III – Confidence intervals calculation
III.1
The standard deviation for file-based analysis
III.2
The standard deviation for condition-based analysis
III.3
Exceptional cases
Appendix IV – Normality test
Appendix V – Statistical significance of the rmse_tot* across all
experiments
Bibliography