Real-time Presence Calibration in Mixed Reality Enabled by Reaction Time Measurement

Tuesday - November 28 2023, 17:57 UTC - 11 months ago

UMass Amherst researchers have identified reaction time as a reliable measure of presence within mixed reality, allowing for the calibration of users in real time. Uses of mixed reality such as in the healthcare field have implications for optimizing the program for better user experiences. Other applications of mixed reality such as in the gaming and construction industry have also seen its potential.

In the real world/digital world cross-over of mixed reality, a user's immersive engagement with the program is called presence. Now, UMass Amherst researchers are the first to identify reaction time as a potential presence measurement tool. Their findings, published in IEEE Transactions on Visualization and Computer Graphics, have implications for calibrating mixed reality to the user in real time.

"In virtual reality, the user is in the virtual world; they have no connection with their physical world around them," explains Fatima Anwar, assistant professor of electrical and computer engineering, and an author on the paper.

"Mixed reality is a combination of both: You can see your physical world, but then on top of that, you have that spatially related information that is virtual." She gives attaching a virtual keyboard onto a physical table as an example. This is similar to augmented reality but takes it a step further by making the digital elements more interactive with the user and the environment.

The uses for mixed reality are most obvious within gaming, but Anwar says that it's rapidly expanding into other fields: academics, industry, construction and health care.

However, mixed reality experiences vary in quality: "Does the user feel that they are present in that environment? How immersive do they feel? And how does that impact their interactions with the environment?" asks Anwar. This is what is defined as "presence".

Up to now, presence has been measured with subjective questionnaires after a user exits a mixed-reality program. Unfortunately, when presence is measured after the fact, it's hard to capture a user's feelings of the entire experience, especially during long exposure scenes. (Also, people are not very articulate in describing their feelings, making them an unreliable data source.) The ultimate goal is to have an instantaneous measure of presence so that the mixed reality simulation can be adjusted in the moment for optimal presence. "Oh, their presence is going down. Let's do an intervention," says Anwar.

Yasra Chandio, doctoral candidate in computer engineering and lead study author, gives medical procedures as an example of the importance of this real-time presence calibration: If a surgeon needs millimeter-level precision, they may use mixed reality as a guide to tell them exactly where they need to operate.

"If we just show the organ in front of them, and we don't adjust for the height of the surgeon, for instance, that could be delaying the surgeon and could have inaccuracies for them," she says. Low presence can also contribute to cybersickness, a feeling of dizziness or nausea that can occur in the body when a user's bodily perception does not align with what they're seeing. However, if the mixed reality system is internally monitoring presence, it can make adjustments in real-time, like moving the virtual organ rendering closer to eye level.

One marker within mixed reality that can be measured continuously and passively is reaction time, or how quickly a user interacts with the virtual elements. Through a series of experiments, the researchers determined that reaction time was a reliable measure of presence within the mixed reality environment.