Towards Unified Standards for Replicable and Generalizable User Studies in Extended Reality

Abstract

The number of paper submissions on XR systems and interaction techniques has risen drastically in the last years. While this positive trend reflects a growing and vibrant research community, the academic system also increasingly incentivizes the rapid publication of research findings to obtain a degree, secure funding, and get promoted. Especially for early-career researchers, this creates a tension between producing valid and reliable results for one project and moving on to the next. This carries the risk that one might, voluntarily or involuntarily, reach conclusions that are neither generalizable nor reproducible.

In this talk, we take a closer look at the heart of decision-making in XR interaction design: the realization and evaluation of empirical user studies, where many seemingly minor design decisions can influence the overall conclusions drawn from the study. We look at selected quality criteria contributing to the reproducibility of empirical findings and interactively discuss minimum standards for future user studies to be published in the top-tier conferences of our field.