The Importance of UX Research: Defining VR Evaluation Metrics

Andi Mastrosavas

As organisations mature their data and insights capabilities, one area, often left behind, is UX metrics. A business may acknowledge the ROI of UX and welcome the qualitative complement to the quantitative data pouring in from their products yet still lag when it comes to quantifying the user experience and its impacts. An indicator that your organisation is at this stage is when research and design activities, while valued, sit outside the product development cycle.

When UX becomes better integrated, an organisation’s success metrics, the ones most highly valued by executives, focus on user experience outcomes. It is only at this stage of maturity that efforts can support and contribute to industry standards.

While championing a product led transformation in the VR space, I had the opportunity to define and validate some standards around VR evaluation metrics specifically tied to commerce. Those metrics became the leading indicators of success for the VR experiences we created.

VR as a Merchandising Medium

It’s no secret that it can be hard to evaluate physical products online effectively. For example, you purchase that perfect ottoman to go with your sofa only to find it’s much larger than it appeared in photographs. So to help consumers make confident purchases, E-commerce evolved to include interacting with 3D models, providing a better experience than traditional 2D images could.

But that interaction was limited and lacked contextual cues spatially. Increasingly, immersive experiences, like virtual reality (VR), are deployed to solve various consumer problems. VR has emerged as a useful medium for exploring products that benefit from spatial environments. Why research furniture online or in-store when you can evaluate it virtually in the context of your own home or aspirationally elsewhere.

The Challenge

Historically, VR was not developed for merchandising physical products. This meant that an evaluative approach for this type of innovation was lacking, and benchmark data was virtually non-existent. Therefore, a new combination of measurements needed to be defined to develop evaluation metrics to ensure the technology would be adopted and predictive of revenue growth.

The Approach

VR is unique in that it combines both hardware and software components to receive input from the user and convey sensory outputs to create the illusion of a virtual world.

A common issue with all immersive extended reality environments, such as VR, is cybersickness. When the vestibular system suggests the body is stationary but the ocular system signals to the brain that the body is in motion, as can happen when using locomotion interactions in VR, these conflicting communications create cybersickness. Common symptoms include nausea, eyestrain, headaches, disorientation, and dizziness. Therefore, measuring for and eliminating cybersickness is crucial.

On the other end of the spectrum is immersion or sense of presence, defined as the experience of being present in one environment while being physically situated in another. An integral part of VR success relies on the consumer believing the reality of the world created around them. Therefore, a sense of presence is critical for VR effectiveness.

Not unique to VR, but essential for all systems that humans interact with, is ease of use. The mental models required to translate interactions with physical controllers or hands to actions taken in the virtual world generate a high cognitive load. It is vital to reduce this to enable the consumer to focus on the products they are evaluating rather than the system they interact with.

Combined, these factors either help or hinder a consumer from making a purchase decision and result in quantifiable customer satisfaction. This led to an hypothesis about what to measure.

The Hypothesis

If

  • The virtual environment provides a high level of immersion:
    • (Sense of Presence: consumers are sufficiently immersed in the virtual environment to believe the reality of the world created.)

And

  • Interaction paradigms are intuitive and easy to use:
    • (Ease of Use: consumers can easily navigate the virtual world and interact such that it strengthens rather than detracts from immersion.)

And

  • The VR technology is comfortable for the consumer:
    • (User Comfort: the human factors of virtual environments, wearing headsets, and using physical controllers do not impact the experience.)

Then:

  • The experience will increase customer satisfaction and the likelihood of a purchase:
    • (Satisfaction: consumers recommend the experience to others as a useful medium for evaluating physical products in context.)

The Instruments

The next phase was to apply reliable and valid instruments for measurement. Instruments were selected due to their reliability and extensive use over many decades.

Ease of Use:

Depending on the stage of development and the type of usability test performed, a different Ease of Use instrument was deployed. The System Usability Scale (SUS) is a reliable tool for measuring the usability of a system overall. In contrast, the Single Ease Question (SEQ) is a simple, reliable measure for understanding task-based Ease of Use. The After Scenario Questionnaire (ASQ) is a more robust version of the SEQ, and the Subjective Mental Effort Questionnaire (SMEQ) measures the mental effort involved in a particular task. It is a reliable method that correlates highly with SUS scores and completion time, completion rates, and errors.

Sense of Presence:

Similarly, with Sense of Presence, depending on the level of fidelity of the technical art, either the Slater, Usoh & Steed presence questionnaire (SUS-PQ) or the Witmer & Singer (PQ) version was used. The SUS-PQ is the most influential and widely used survey instrument for measuring presence in a virtual environment. In comparison, the PQ is more robust and time-consuming to gather the data and analyze.

User Comfort:

User Comfort utilised a modified version of the simulator sickness questionnaire (SSQ), initially developed for military flight simulators and widely adopted to quantify the extent and severity of simulator sickness symptoms elicited by modern virtual reality environments. It measures sickness across three subscales; nausea, oculomotor & disorientation.

Satisfaction:

Satisfaction was measured by Net Promoter Score (NPS), the most widely used and accepted measure for customer experience satisfaction, and as a predictor of business growth.

Instrument Summary

 

Ease of Use Sense of Presence User Comfort Satisfaction
?      SUS

?      SEQ

?      ASQ

?      SMEQ

?      SUS-PQ

?      PQ

?      SSQ ?      NPS

The Results

While I can’t share the data, what I can share is that the hypothesis was validated. As Ease of Use, User Comfort, and Sense of Presence increased, so did Customer Satisfaction. While Ease of Use has been linked to Customer Satisfaction metrics across other mediums, this is the first time I have seen it demonstrate the correlation in VR.

Interestingly, and unique to VR, we also found a correlation between User Comfort and Sense of Presence where discomfort negatively impacted the level of immersion. In addition, we also observed a correlation between Ease of Use and Sense of Presence, where the more intuitive the interactions were, the higher the perceived immersion.

These evaluation metrics and instruments, when combined, provided a robust way to evaluate throughout the development cycle to ensure all aspects of the experience were helping and not hindering the consumer from making a confident purchase decision in a new and largely untested medium.

Wherever you are in your UX metrics journey, it’s always a good time to re-evaluate. How is the user experience impacting your product? How do you intend to measure its impacts? And what changes do you need to make in your UX and Product practice to help centre your organisation around it to deliver innovation that resonates in the market regularly?

Want more insights into different research methods. Check out Andi’s blog on the top 5 best practices for observational research.

Andi Mastrosavas

Andi Mastrosavas | Author

Andi is a Principal Product Consultant at Brainmates, specialising in strengthening product management practices, to deliver value to customers with products that outperform in the market. Andi has a Master of Science in Human Computer Interaction and a Master of Science in Global Technology & Development. If you are interested in learning more about how you can apply mixed methods to maximise the impact of your research, or anything else product related, you can reach her via andi@brainmates.com.au.

Product Management Training

Product Women

Product Women

Inspirational, authentic, fun! An awesome space to build your career, meet great people and learn.

Learn More