Page 25 - Methodology for measurement of Quality of Service (QoS) Key Performance Performance Indicators (KPIs) for Digital Financial Services

– Location Log Sheet: Initial, intermediate and final In case of fully automated data acquisition, typical
checks on device set-up and status. time resolution is 1msec while typical phase durations
are at least a couple of 100msec or longer.
– Data Log Sheet (P2P Transfer): Acquisition of results The other extreme is entirely manual time-taking
for service tests.
where time resolutions are much longer, typically 1 s
– Event Log Sheet: capture of unusual conditions or or even more considering that times have to be read
events during tests. from a display which by itself may contain additional
An example of an actual campaign log used in the pilot delay. Even in the case of semi-automatic data acqui-
campaign done in Ghana is shown in Appendix I sition where some kind of stopwatch with high reso-
lution is used, human reaction time and its jitter will
6.5 Handling of confirmation/information SMS result in an effective time resolution in the order of
(secondary information) some 100 msec.
This data shall be retrieved from the device at least once This means that a fine-grained time recording as in-
per day, and transmitted/uploaded to a target destina- dicated by Figure 4-2 will not be possible and use case
tion (typically by email). modelling will have to be restricted to main phases.
For retrieval, several tools are available. For the pilot From a practical point of view, this will be the overall
project, the app “SMS Backup & Restore” has been se- transaction time from invocation of the MoMo service
lected. The app copies local SMS to a XML file. This file to its completion (end to end duration), and the core
can be stored locally and be sent to a remote destina- transaction time, i.e. the time between triggering the
tion via e-mail. transfer after all input information has been provided,
The following sub clause describes the model set-up and its completion.
of the app. Data acquisition may deliberately be done fully man-
After the data has been successfully uploaded, it can ual, or points of observation to obtain trigger point
be deleted on the device. The data file just uploaded events may be limited. . In such a situation, some of the
can be moved to a backup storage location. Until then, generic KPI, as described in clause 5, are not applicable
the data shall be kept on the device as a back-up copy. due to the reasons described. The following set of prac-
tical KPI can be used:
■ NOTE: If the devices are restricted in functionality In all cases, a valid sample requires that all trigger
(e.g. to act as “Feature phones”, transfer via e-mail points used in computation are valid, i.e. present. Indi-
requires that these restrictions (e.g. “no mobile cators of type ‘time’ are therefore computed from trans-
data”) are removed for the transfer. It is important actions where respective phases have been completed
to re-establish the correct settings for DFS testing successfully.
afterwards, or prior to a new set of tests. For the overall completion times, the E2E version us-
ing T1 was selected although it includes times for manu-
al activity. Reasoning is as follows: A KPI, as an indicator
7 SPECIAL CONSIDERATIONS FOR MANUALLY expressing the end-user perspective, should provide a
OPERATED TESTING AND TIME-TAKING
realistic estimate of a service’s behavior. Manual activ-
ity is an integral part of service usage and therefore it
The considerations described so far assume that makes sense to include respective times into an indica-
time-taking provides a precision of time measurement tor. Assuming that a testing team can be compared to
which is sufficiently higher than typical times for re- an experienced user, times taken by such a team can be
spective phases of the event flow.
viewed as a valid estimation of manual components of
service usage.

TABLE 7-1: Timestamps (trigger points) used for the practical KPI

SYMBOL DESCRIPTION
T1 Start of transaction (activation of the DFS function/application on the device)
T2 All necessary input data has been entered and the actual money transfer is triggered.
One of T3 Reception of the primary success criterion (information about the successful completion of the transaction), or
T4 Reception of an information stating that the transaction has failed
T5 Time-out limit reached without a positive or negative reaction from the service
T6 Reception of the summary SMS in the A-side mobile device
T7 Reception of the summary SMS in the B-side mobile device

Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 23Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 23