Page 79 - Kaleidoscope Academic Conference Proceedings 2022

Extended reality – How to boost quality of experience and interoperability

ten statements about the user-friendliness of an app, which 7. DISCUSSION AND LESSONS LEARNED
test subjects can agree or disagree with. One vote is given for
each statement on a Likert scale ranging from one (strongly This section discusses the approach of the framework and
disagree) to five (strongly agree) [16]. Brooke’s proposed presents some key lessons learned during the research work.
questions were slightly customized and are shown below:
7.1 Discussion
1. I think that I would use the AR application
frequently to get information about my car. The proposed framework describes the implementation for
2. I found the AR application unnecessarily complex. model-based use cases with multiple variants. In contrast to
3. I thought the AR application was easy to use. other previous referred approaches, this framework provides
4. I think that I would need the help of a technical state of the art information about the implementation with
person to use or understand the AR app. Vuforia MTs and the handling of a variety of models. The
5. I found the various functions of the AR application usage of AssetBundles is only one possible solution, but it is
most suitable for the given use case.
were well integrated.
6. I thought that the AR application still has many The evaluation regarding usability is not representative at
bugs and doesn’t work properly. this stage of work, because the number of participants is not
7. I would imagine that most people would learn to sufficient, whereby the application is still in an infantile
use the AR application very quickly. state. Nevertheless, a first impression on usability may be
8. I found the AR application very cumbersome to use. contained and that downloading models during runtime has
9. I felt very confident using the AR application. no negative influence on it. The performance evaluation
is given by time, whereas the values strongly depend on
10. I needed to learn a lot of things before I could use Internet speed, end device and model size. The results cannot
the AR application. be referenced or compared to other frameworks but can be
classified subjectively in relation to the requirements of the
The result of the SUS is a value in a range from 0 to 100. The use case.
questions are structured in such a way that in the best case,
the surveyed alternately ticks the rightmost (odd-numbered 7.2 Lessons learned
questions) and then the leftmost (even-numbered questions).
For each question, a score between zero and four is calculated. During the development process we faced multiple
The sum of all scores is multiplied with 2.5 to get the final challenges and learned a few lessons.
SUS score. The following rules apply to the calculation of
the questions: • We started developing without any previous experience
• Odd-numbered questions: Score= arithmetic mean-1 in AR and only basic programming skills. Especially for
novices, it is important to have fundamental blueprints
• Even-numbered questions: Score= 5-arithmetic mean and best practices. There is a lot of literature about
specific use cases, but there is a scarcity in fundamental
The usability of the final prototype was evaluated with and generic approaches for design, implementation and
Brooke’s SUS. The application uses AR to display evaluation of model-based AR applications.
context-information about multiple car models. We
conducted the experiment with 11 test subjects, who were • When it comes to practice, generic approaches come to
asked to test the prototype and answer ten questions with their limits due to multi-variant products. In the
predefined answers via an online survey. The participants do automotive industry, offering customer-specific
not represent the existing diversity of age, gender and user variations has been standard since the early 1990s.
behavior, because at this stage of preliminary work it was not These must be implemented as efficiently as mass
necessary to provide representative results. Nevertheless, it is production which also includes an efficient creation of
worth mentioning that no testers had previous experience with instruction material.
an AR instruction application.
• A car always has different series and customizations, and
We calculated the SUS score of our application with the each of them is another variant of the original car with
results from the survey according to the above explained another instruction. Each variant must be implemented
method and obtained an SUS-value of 90. To classify this and included in the instructional application which leads
value, a grading scale presented by Lewis and Sauro was used, to a large amount of data and inefficient performance.
to assign the SUS score a grade and a normalized percentage. Such a handling of the complexity of variants in an AR
They determined an average value of 68 and assigned it the context has not been adequately researched yet.
grade "C" [17]. Comparing the calculated SUS result to the
grading scale, it is rated with the best grade (A+) and therefore
belongs in the percentage range of the best 96-100%.

– 33 –