Page 13 - Methodology for measurement of Quality of Service (QoS) Key Performance Performance Indicators (KPIs) for Digital Financial Services

scheme described in the present document. In Annex 3.1.2 Definition of action flows
D, the basics for design and definition of QoS and QoE A team, which conducts a test, typically consists of two
metrics are described for readers which are not already persons, named P1 and P2. Alternative team sizes (e.g.
familiar with the topic. Annex E gives examples for con- five persons assigned to one of the four testing phones
sistent naming of files and other data items in a cam- and the observer phone) or the option to use more than
paign. Annex F provides an overview table of KPIs and one team per location are for further study. Based on
related trigger points. experience made so far, it appears that any such solu-
Furthermore, appendices provide specific informa- tion should be accompanied by increased tool support
tion on the pilot testing campaign itself, which has been (such as partly automated time-taking as described in
performed in Ghana in the first part of 2018. Appendix clause 10.2) in order to manage the increased level of
I shows the log sheets used which can serve as a tem- work intensity and to maintain high data quality.
plate for similar campaigns. Appendix II, provides an This team, acting in the roles of A and B party, re-
overview of the testing campaign in Ghana itself, while spectively will do a single transfer.
Appendix III shows how a component of data sourcing, In parallel to the actual transfer action, the person
a specific tool for backing up notification SMS, is set up. designated as P2 also operates the observer phone
Please note that the current document only covers (since P1, in the A party role, is engaged with perform-
the methodology for tests done from an individual us- ing the transfer while P2 in the B party role is mostly idle
er’s (end to end) perspective, acting within a given DFS with respect to the money transfer).
ecosystem under current load conditions. It may be de- A cycle of transfers consists of four (4) transactions,
sirable to extend the scope of testing to capacity tests, using all combinations of smartphones and feature
which would involve creation of defined load to a DFS phones assigned to the A and B party roles. After this
ecosystem in order to determine the robustness of DFS cycle, the money transferred (less operators’ charges) is
functionality under these conditions. Such extensions again available on SP1 and FP1, respectively.
can be easily created from the given methodology.
Their execution is mainly a matter of scale of required 3.2 Neutral starting state
resources. A particular property of systematic service tests is a fre-
quency of service usages which is significantly higher
than the usage frequency created by a typical end user.
3 TEST SCENARIO UNDER CONSIDERATION A testing campaign, therefore, should contain systemat-
ic tests to make sure that usage frequencies typical for
In the following, the use case “Person-to-Person” (P2P) testing do not affect testing results with respect to the
money transfer is described. The methodology is de- end-user perspective. A possible way to realize this can
signed to be easily extended to other use cases in future be systematic tests, which are run prior to the actual
projects. campaign and where the testing frequency is varied.
There are two categories of effects. Firstly, as for ser-
3.1 Definitions vices other than DFS, after each service usage a certain
relaxation or guard time is expected to be required. This
3.1.1 Roles, entities and abbreviations
is well understood, e.g. for guard times after telephone
TABLE 3-1: Terms and abbreviations specific to connections have ended — in order to enable the ser-
descriptions of DFS test cases vice to reach its neutral state again. This first category
is considered to be uncritical as such guard times are
A Party and Formal roles for transfer, e.g. A (active role) typically in the range of 10 to 30 seconds. Depending
B Party transfers money to B (passive role).
on the respective implementation, a second category of
SPx and FPx Designation of device types used for a effects may, however, exist which concerns longer peri-
transfer, Smartphone x, Feature phone x.
ods of time.
Dx More general indexed device For example, some kind of local memory (e.g. like
description (e.g. D1, D2…)
a web browser’s cache) where content already down-
OPx Observer Phone x loaded will be kept for a longer period of time or even
Px Person x, designation of a tester/ for an indefinite time may be involved.
operator (independent of role) The request for the same content at a later point
■ NOTE 1: The testing methodology comprises typically of a round-trip in time would then access the local cache, directly in-
transfer in definite N steps. As a result, thereof, roles between devices stead of triggering an actual data transfer, and therefore
and operators are switched after every transaction. would massively impact the correctness of the results
■ NOTE 2: In order to optimize testing efficiency and to minimize the of the test campaign (by showing quite short transfer
risk of errors during the test preparation, the assignment of devices
to accounts should be fixed. Consequently, the assignment of roles is delays).
being switched between the devices in a cyclic manner (with Smart According to findings of respective pre-tests, ap-
Phone to the left and Feature Phone to the right of a Person during propriate steps should be taken (such as clearing local
manual tests). This is to ensure that the manual cycle of tests is uni-
form to the transaction cycle as illustrated in Table 3 1. memories). As long as effects are quantitative rather

Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 11Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 11