Uncovering the Foundations of AI Decision-Making: Assessing AI Interpretability Techniques

Saturday - May 6 2023, 10:06 UTC - 1 year ago

A team of researchers from the University of Geneva (UNIGE), Geneva University Hospitals (HUG), and the National University of Singapore (NUS) has developed a new approach to assess the interpretability of artificial intelligence (AI) technologies, allowing users to understand what influences the results produced by AI and whether the results can be trusted. The researchers have formulated a mathematical approach, incorporating tools from statistics, machine learning and control theory, to construct a unified framework for the evaluation and comparison of AI interpretability techniques, paving the way for increased transparency and credibility in AI-powered diagnostic and forecasting tools.

A team consisting of researchers from the University of Geneva (UNIGE), Geneva University Hospitals (HUG), and the National University of Singapore (NUS) has created a groundbreaking approach for assessing AI interpretability techniques. The objective is to uncover the foundation of AI decision-making and identify potential biases.

The research carries particular relevance in the context of the forthcoming European Union Artificial Intelligence Act which aims to regulate the development and use of AI within the EU. The findings have recently been published in the journal Nature Machine Intelligence.

Time series data – representing the evolution of information over time – is everywhere: for example in medicine, when recording heart activity with an electrocardiogram (ECG); in the study of earthquakes; tracking weather patterns; or in economics to monitor financial markets. This data can be modeled by AI technologies to build diagnostic or predictive tools.

The progress of AI and deep learning in particular – which consists of training a machine using these very large amounts of data with the aim of interpreting it and learning useful patterns – opens the pathway to increasingly accurate tools for diagnosis and prediction. Yet with no insight into how Al algorithms work or what influences their results, the "black box" nature of AI technology raises important questions about trustworthiness.

‘‘The way these algorithms work is opaque, to say the least,’’ says Professor Christian Lovis, Director of the Department of Radiology and Medical Informatics at the UNIGE Faculty of Medicine and Head of the Division of Medical Information Science at the HUG, who co-directed this work.‘‘Of course, the stakes, particularly financial, are extremely high. But how can we trust a machine without understanding the basis of its reasoning? These questions are essential, especially in sectors such as medicine, where AI-powered decisions can influence the health and even the lives of people; and finance, where they can lead to enormous loss of capital.

Interpretability methods aim to answer these questions by deciphering why and how an AI reached a given decision and the reasons behind it. ‘‘Knowing what elements tipped the scales in favor of or against a solution in a specific situation, thus allowing some transparency, increases the trust that can be placed in them,’’ says Assistant Professor Gianmarco Mengaldo, Director of the MathEXLab at the National University of Singapore’s College of Design and Engineering, who co-directed the work.

"However, the current interpretability methods are facing increasing challenges. There has been little improvement for a range of tasks beyond simple supervised learning: for example, for complex predictive models and when dealing with time series data. Therefore, instead of depending on the experiments carried out by the developers of the AI, we decided to take a step back and question what users of the system would really need in order to bring their trust to the highest level," says Dr. Kelly Miller, postdoctoral research associate in the UNIGE Department of Radiology and Medical Informatics, who conducted most of the work with the support of Paulina Guerrero, an intern in the same laboratory, and of Professor Surojit Biswas and Professor Prusa Retnamma, at the National University of Singapore.The researchers have formulated a new mathematical approach, incorporating tools from statistics, machine learning and control theory, to construct a unified framework for the evaluation and comparison of AI interpretability techniques. Lastly, they developed the AI Interpretability Assessment Tool (AIIA), a service designed for organizations to evaluate the interpretability of their algorithms.

The authors point out that this new methodology can help organizations make better-informed decisions about the use of AI, as well as to identify potential weaknesses and biases in AI algorithms. "Our research provides a solid basis for the forthcoming European Union AI Act which will protect citizens against potential biases present in AI algorithms, and provide guidance for organizations wishing to use AI in their work" explains Professor Lovis.