Page 35 - Methodology for measurement of Quality of Service (QoS) Key Performance Performance Indicators (KPIs) for Digital Financial Services

TABLE D-1: Expressions used in definitions of KPI computation
N(EI) NUMBER OF EVENTS WITH INDEX I.
T(Ei, Ej) Time elapsed between the events with indices I and j. This quantity applies to one particular transaction and is
only valid if both events are present.
A KPI of ‘time’ type is the average (usually arithmetic mean) of respective transaction-wise T values.
R(Ei, Ej) Rate (percentage) of events with index I with respect to events j.
Typically, Event index j represents a “try” and index I the related success indicator for a given phase. In that case,
R stands for the success rate of the given phase.
This expression is only valid if N(Ej) > 0 which means that for a valid rate indicator there must be at least one ‘try’
occurrence of the respective phase.
Technically, the condition N(Ei) <= N(Ej) is also met. This is however a technical cross-checking condition which
is assumed to be fulfilled always if the underlying measurement and processing mechanism is properly defined
and functioning.

D.4 UNDERSTANDING OF KPI Even though DFS appears primarily to be a direct, in-
teractive type of service, it has some store-and-forward
Reported KPI represent the results of respective mea- properties. This relates to the matter of using reason-
surements. Under the assumption that a statistically sig- able values for time-outs.
nificant number of samples has been taken, they also Here, several aspects have to be considered carefully.
represent a prediction on the outcome of tests done If time-out values are too short, this would not only rep-
with the same set of testing conditions, i.e. parameters resent customer perspective by painting a too-negative
of a test.
picture of the service. It would also increase the amount
■ NOTE: In statistics, sample usually refers to a set of of money needed for insertion after assumed failure.
measurements, i.e. the entirety of all data from a giv- From an operational point of view, it would also cre-
en test. For the purpose of this document, sample ate additional complexity. If time-out is declared due to
5
(singular) denotes a single data point for a KPI com- a missing response of the system, the next transaction
putation, i.e. information related to one particular will be started. This would then either need a dedicat-
transaction. This is equivalent to the term “sample ed cancellation of the ongoing transaction, or the test
point” or “observation” in the statistics context. would be in a kind of undefined state.
Moreover, due to the secondary response (summary
Therefore, the functional descriptions use the term SMS), there is actually a double time frame. The primary
‘probability’ for KPI which have the type of a rate, in ac- confirmation may have arrived, but the summary SMS
cordance with the wording in ETSI TS 102 250-2 and are still under way. It is assumed for the time being—with
Rec. ITU-T E.804. a note that this should be validated—that these SMS are
The term ‘time’ is used in two ways. If the context is actually decoupled from the DFS process. If the waiting
individual transactions, it means respective single val- time for these SMS has expired, and the next transac-
ues for that particular transaction. In a KPI context, it tion is started, they can still appear. The procedure also
designates an aggregated value. If no other definition needs to cover this possibility in order not to introduce
is made, this shall mean the average of transaction-wise confusion in case it occurs.
values. Using long time-outs—to reduce this risk, understood
To avoid duplications of text, validity rules are as- as hoping a transaction without a clear response may
sumed to be generic, i.e. relate to the formal validity turn out to be successful after all—will however reduce
definitions outline in Terminology.
the yield of a measurement campaign in case of a high
actual loss rate.
D.5 SPECIFIC PROBLEMS OF DFS Also, some care needs to be taken in definition of a
TRANSACTIONS clean-up process. A clean-up process should not pose
the risk of messing up the test; i.e. an attempt to roll
DFS is, at least from a technical testing perspective, a back a transaction may not only cause time delay but
store and forward service. For pragmatic purpose, a also create additional disturbance in the system and en-
time-out condition for a test case is necessary; other- danger data integrity. At present it appears to be the
wise, a ‘hanging’ transaction would effectively block a most sensible decision to refrain from any situational
test. roll-back attempts and assume that some final tidy-
Considering that a field test for DFS is transferring ing-up is made. For test design this means that suffi-
real money, there is the basic question of clean-up. In cient reserves—and a good monitoring—have to be allo-
case a transaction is unsuccessful, the money involved cated to keep the testing process going.
in this transaction would have to be assumed, and new
money would have to be inserted into the loop.

Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 33Methodology for measurement of Quality of Service (QoS) Key Performance Indicators (KPIs) for Digital Financial Services • 33