Understanding Artificial Intelligence Predictions In Augmented Reality (XAIR Framework)

Tuesday - May 9 2023, 13:03 UTC - 1 year ago

Meta Realilty Labs recently developed the XAIR Framework, a framework designed to help developers make artificial intelligence (AI) and machine learning tools easier to understand by providing more effective explanations for AI predictions in augmented reality (AR) scenarios. XAIR is the first framework of its kind to facilitate the design of explainable AI for AR applications, which it does so by identifying five key factors that determine the design of the 'when, what, how' aspects.

While artificial intelligence (AI) and machine learning tools are now commonly used to enhance technological applications, the underpinnings of many of these tools are hard to decipher. This is because most of them are based on 'black box' models, models that analyze data and learn to make predictions about it but that do not share the processes behind these predictions with human users.

Researchers at Meta Reality Labs recently created XAIR, a framework that could help developers to make the processes underpinning the predictions of AI easier to understand. This framework, introduced in a paper presented and published as part of the Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, is specifically designed to create explainable AI (XAI) systems that can be applied in various augmented reality (AR) settings.

"As black-box models are increasingly being employed in daily life, we are having more and more concerns about humans misusing AI and losing control," Xuhai Xu and Anna Yu, two of the researchers who carried out the study, told Tech Xplore. "This has led to the need to make algorithms easier to understand, leading to the surge of XAI. Existing studies have found that XAI can help end-users resolve confusion and build trust. Therefore, industrial practitioners have tried to use XAI to improve user experiences." .

AR technology allows users to view a modified version of their surrounding environment, which integrates digital elements, sounds and/or visual enhancements. These 'digitally enhanced' versions of reality can be viewed through head-mounted displays, goggles, other wearable gear and even simply through the smartphone screen.

Recently, some researchers have been exploring the use of AI to enhance AR applications, for instance making them more responsive to changes in a user's environment or allowing them to analyze and make predictions about specific objects. Xu, Yu and their colleagues set out to create a framework that could make the results of these AI tools for AR applications easier to understand, thus increasing users' trust in them.

"As AI models will be needed for context-aware, everyday AR, XAI will also be essential because end-users will interact with all kinds of AI outcomes," Xu and Yu said. "XAI can be useful in many ways, such as making intelligent AR behavior interpretable, resolving confusion or surprise for unexpected AI outcomes, promoting privacy awareness, and building trust. Given the importance of XAI for AR, we aim to answer the research question about the right way to create effective XAI experiences for AR in everyday scenarios." .

The team at Meta created the XAIR framework hoping that it would facilitate the design of XAI for AR applications. Their framework essentially addresses three open questions: when, what and how? Answers to these questions can be used to provide more effective explanations for AI predictions in AR scenarios. In addition to helping developers to create AI that can answer these three questions, XAIR outlines a series of key guidelines for researchers and developers working on XAI for AR applications.

"We identified five key factors based on a large-scale literature review," Xu and Yu explained. "These factors determine the design of the 'when, what, how' aspects, including two timeline factors, two visual factors and one user factor. We illustrate examples of each factor by linking them to the outcomes of our literature review. The XAIR framework provides a valuable tool for creating XAI experiences for AR." .