Exploring the Inner Workings of Neural Network Image Recognition Systems

Saturday - November 18 2023, 12:18 UTC - 1 year ago

A new tool developed at Purdue University makes it easier to trace errors in neural network image recognition. The tool is available on GitHub and is able to catch neural networks mistaking the identity of images, such as mistaking cars for cassette players. The tool helps researchers understand the origin of errors that are obvious to humans.

In the background of image recognition software that can ID our friends on social media and wildflowers in our yard are neural networks, a type of artificial intelligence inspired by how own our brains process data. While neural networks sprint through data, their architecture makes it difficult to trace the origin of errors that are obvious to humans—like confusing a Converse high-top with an ankle boot—limiting their use in more vital work like health care image analysis or research. A new tool developed at Purdue University makes finding those errors as simple as spotting mountaintops from an airplane.

"In a sense, if a neural network were able to speak, we're showing you what it would be trying to say," said David Gleich, a Purdue professor of computer science in the College of Science who developed the tool, which is featured in a paper to published in Nature Machine Intelligence. "The tool we've developed helps you find places where the network is saying, 'Hey, I need more information to do what you've asked.' I would advise people to use this tool on any high-stakes neural network decision scenarios or image prediction task." .

Code for the tool is available on GitHub, as are use case demonstrations. Gleich collaborated on the research with Tamal K. Dey, also a Purdue professor of computer science, and Meng Liu, a former Purdue graduate student who earned a doctorate in computer science.

In testing their approach, Gleich's team caught neural networks mistaking the identity of images in databases of everything from chest X-rays and gene sequences to apparel. In one example, a neural network repeatedly mislabeled images of cars from the Imagenette database as cassette players. The reason? The pictures were drawn from online sales listings and included tags for the cars' stereo equipment.

Neural network image recognition systems are essentially algorithms that process data in a way that mimics the weighted firing pattern of neurons as an image is analyzed and identified. A system is trained to its task—such as identifying an animal, a garment or a tumor—with a "training set" of images that includes data on each pixel, tagging and other information, and the identity of the image as classified within a particular category.

Using the training set, the network learns, or "extracts," the information it needs in order to match the input values with the category. This information, a string of numbers called an embedded vector, is used to calculate the probability that the image belongs to each of the possible categories. Generally speaking, the correct identity of the image is within the category with the highest probability.

But the embedded vectors and probabilities don't correlate to a decision-making process that humans would recognize. Feed in 100,000 numbers representing the known data, and the network produces an embedded vector of 128 numbers that don't correspond to physical features, although they do make it possible for the network to classify the image.

In other words, you can't open the hood on the algorithms of a trained system and follow along. Between the input values and the predicted ide .