Evaluating Virtual Reality Systems

Hello Everyone! Welcome! In this blog article you are given an overview about the topic „Evaluating Virtual Reality (VR) Systems“. It will give you an insight about what VR is, a little information about the history and lastly which evaluation methods are used.

Before we dive into the different virtual reality evaluation methods, let's have a look into the Definition of VR. There are three definitions of about a ton of definitions which caught my interest:

If we look into these three definitions, we can see the similarities in defining virtual reality. All three definitions contain the word googles and gloves and a electronically simulated environment. They define the term virtual reality in association with the technological hardware. Beside this definitions there are some, which give you definitions about the human experience. Having someone immerse into a virtual world and giving him the opportunity to interact with this world, while using specific hardware. (Stuer, 1992) In Addition to the here are the important three „I“´s of VR:

_{This image was retrieved from: https://www.researchgate.net/figure/Virtual-reality-triangle-Immersion-Interaction-Imagination-BL93_fig2_281047284}

In the table below is a short history of the VR shown. The data was taken of the paper from Zhao Q. (2009).

First of all the most important questions for researches are like: What are the user needs? How will the user behave in using the product or system? These are the kind of questions a user experience researcher seeks answers for. This answers are very important for the improvement of the products or systems. Also it always depends what the researches seek for. Do they want the system to be tested? Or do they want to get information about the user feelings, while using the system?. (Murtza, Monroe&Youmans, 2017)

When we think about usability evaluation methods the first method that comes in mind is the heuristic evaluation of Nielsen (1993). Beside that there are other methods out in the evaluation field like the co-operative evaluation of Monk et al. (1993) with users to find problems or the cognitive walkthrough of Wharton et al. (1994). All of these methods are providing a guidance for evaluation. But they are not perfect and are not suitable for solving every problem. They do lack some information about for example manipulating objects, navigating in virtual worlds etc. (Sutcliffe and Kaur, 2008) So Sutclifffe and Kaur (2008) developed a walktrough evaluation, where they expanded the Wharton et al. (1994) evaluation by proposing Norman's (1986) mdoel. This method looks like this:

• Prepare scenarios that represents the users tasks as goals and actions to be carried out, general exploration and navigation, and reaction to system actions.
• Walk through each scenario using the appropriate model, asking the questions detailed in the following sections.
• Critique the design using the question and the GDPs linked to the action stage; record problems in terms of missing GDPs or VE design features.
• Continue walkthrough to the end of the scenario script and having tested all the scenarios, collate the indicated problems into a list of missing GDPs linked to VE design features where possible, and indicate missing features.
• Prioritise problems by subjective judgement of their severity.

Walkthrough Methods are a good guide in terms of evaluation methods for VR Systems, but they do not offer a solution for a thorough analysis.

A.G. Sutcliffe and K. Deol Kaur. 2008. A Usability Evaluation Method for Virtual Reality User Interfaces.

Coates, G. (1992). Program from Invisible Site—a virtual sho, a multimedia performance work presented by George Coates Performance Works, San Francisco, CA, March, 1992.

Greenbaum, P. (1992, March). The lawnmower man. Film and video, 9 (3), pp. 58-62.

Krueger, M. W. (1991). Artificial reality (2nd ed.). Reading, MA: Addison-Wesley.

Monk, A., WRIGHT, P., HABER, J. and DAVENPORT, L. 1993, Improving Your Human ± Computer Interface (Englewood CliVs, NJ: Prentice Hall).

Nielsen, J. 1993, Usability Engineering (New York: Academic Press).

Norman, D. A. 1986, Cognitive engineering. In D. Norman and S. Draper (eds), User Centered System Design: New Perspectives on Human Computer Interaction (Hillsdale NJ: Lawrence Erlbaum Associates), 31 ± 62.

Rabia Murtza, Stephen Monroe, and Robert J Youmans. 2017. Heuristic Evaluation for Virtual Reality Systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 61. SAGE Publications Sage CA: Los Angeles, CA, 2067–2071.

Steuer, J. (1992). Defining virtual reality: Dimensions determining telepresence. Journal of communication, 42(4), 73-93.

Wharton, C., REIMAN, J., LEWIS, C. and POLSON, P. 1994, The cognitive walkthrough method: a practitioner’s guide. In J. Nielsen and R. L. Mack (eds), Usability Inspection Method (New York: Wiley), 105 ± 140.

Zhao, Q. (2009). A survey on virtual reality. Science in China Series F: Information Sciences, 52(3), 348-400.