Page 29 - U4SSC KPIs Verification Manual - A guide for verifiers

5.3.2 Activities involved in assessing “Reasonableness”:

To arrive at your conclusions about the “reasonableness” of the data presented, your judgements
must be supported by research and investigation, and some basic mathematical checking.

Research and/or investigation typically involves searching for data sources about the city in
question. For example, you may question data related to sanitary sewer hookups. Perhaps the city
has cited data from its utility organization. The reported data may appear better than you intuitively
perceived. Does the city-owned utility only extrapolate data related to formal households? What
is the proportion of households within the city that may be informal? This is a good question for
the data compiler from the city that presented the information. It is also an opportunity for you to
do some independent research to learn what other reporting or tacking bodies may have to say
about the subject.

Basic mathematical checking will be required whenever, as you are scanning the data, you discover
that a ratio makes no sense, at the order of magnitude level, based on other reported data. This
could be a simple error on the part of the city or an attempt to obfuscate reality.

In either of the above scenarios, you are not the character judge; you are merely the person
determining if the data presented look reasonable.

If there are any data points where your concerns prevent you from concluding that an input is
“reasonable”, the question must be presented back to the city in your review report.

Of primary importance in reviewing the reasonableness of data, as outlined above, are:

I. A review of the data sources cited with the KPI. They should match the guidelines provided.
Check the data source wherever practical and accessible. Ensure the data presented are
consistent with the source. There is much information to be gleaned via the links provided in
the “source” section of each KPI. While you will not be able to review all the sources in every
city data set you review, you ought to learn and study as much as you can. This will help your
efficiency, confidence and credibility over time.
II. Review the basic maths used in the “Methodology”. Ensure consistency of numerators or
denominators where such represent a city-wide statistic.

III. Consider additional Verifiers’ guide notes as may exist. The guidance contained in the Verifier
Guide section is based on experience learned to date. If you have learned something that you
feel may benefit the guide, note it in your report, ensuring that you are specific as to the KPI
number, classification and dimension. The Verifier Guide is a repository for intelligence garnered
over time from Verifier activities and lessons learned. Such information will be harnessed and
used as a basis for updating this guide.

U4SSC KPIs Verification Manual-A guide for verifiers 19