From Soviet-Built Computer to AI Researcher: The Story of Alexei Efros

Thursday - October 26 2023, 18:19 UTC - 1 year ago

Alexei Efros has had a long journey from using a Soviet-built personal computer as a teenager to becoming a computer scientist and AI researcher at the Berkeley Artificial Intelligence Research Lab. His current work focuses on understanding the wholly visual patterns left behind by AI and improving computer vision technology. He believes that machines still see differently than people, and that it is important to fill data sets with labels and other information in order to make AI function more accurately.

When Alexei Efros moved with his family from Russia to California as a teenager in the 1980s, he brought his Soviet-built personal computer, an Elektronika BK-0010. The machine had no external storage and overheated every few hours, so in order to play video games, he had to write code, troubleshoot, and play fast — before the machine shut down. That cycle, repeated most days, accelerated his learning.

"I was very lucky that this Soviet computer wasn’t very good!" said Efros, who laughs easily and speaks with a mild Russian accent. He doesn’t play as many games nowadays, but that willingness to explore and make the most of his tools remains.

In graduate school at the University of California, Berkeley, Efros began hiking and exploring the Bay Area’s natural beauty. It wasn’t long before he began combining his passion for computers with his enjoyment of these sights. He developed a way to seamlessly patch holes in photographs — for example, replacing an errant dumpster in a photo of a redwood forest with natural-looking trees. Adobe Photoshop later adopted a version of the technique for its "content-aware fill" tool.

Now a computer scientist at the Berkeley Artificial Intelligence Research Lab, Efros combines massive online data sets with machine learning algorithms to understand, model and re-create the visual world. In 2016, the Association for Computing Machinery awarded him its Prize in Computing for his work creating realistic synthetic images, calling him an "image alchemist." .

Efros says that, despite researchers’ best efforts, machines still see fundamentally differently than we do. "Patches of color and brightness require us to connect what we’re seeing now to our memory of where we have seen these things before," Efros said. "This connection gives meaning to what we’re seeing." All too often, machines see what is there in the moment without connecting it to what they have seen before.

But difference can have advantages. In computer vision, Efros appreciates the immediacy of knowing whether an algorithm designed to recognize objects and scenes works on an image. Some of his computer vision questions — such as "What makes Paris look like Paris?" — have a philosophical bent. Others, such as how to address persistent bias in data sets, are practical and pressing.

"There are a lot of people doing AI with language right now," Efros said. "I want to look at the wholly visual patterns that are left behind." By improving computer vision, not only does he hope for better practical applications, like self-driving cars; he also wants to mine those insights to better understand what he calls "human visual intelligence" — how people make sense of what they see.

Quanta Magazine met with Efros in his Berkeley office to talk about scientific superpowers, the difficulty of describing visuals, and how dangerous artificial intelligence really is. The interview has been condensed and edited for clarity.

How has computer vision improved since you were a student? .

When I started my Ph.D., there was almost nothing useful. Some robots were screwing some screws using computer vision, but it was limited to this kind of straightforward task. In the roughly 25 years since then, computer vision has advanced to the point where a large fraction of the start-up or the tech industry is using computer vision in every kind of product. We can do a lot of things now: We can understand what’s in an image, we have some abilities to recognize things in videos. We are making the computer more and more “intelligent.” I believe the reason for this fast development is that a lot of people are looking to fill those same holes that a lot of us see and keep asking the same questions: “Can I do this? Yes, I can do it.” It’s a process of filling the holes, not only with algorithms, but also with data.

Computer vision is limited in part by its reliance on existing data sets. For example, imagine a dataset of images of cats, each with its own label attached to it. If the dataset contains a lot of labels for cats looking to the left, then it probably doesn’t have a lot of images of cats looking to the right. If the dataset doesn’t have a lot of images of cats looking to the right, then it won’t be able to recognize cats looking to the right. As a result, there are biases in images: Some cats are more common than others, or some scenes tend to come up more often. We have to be aware that in many cases these models don’t work everywhere in the world.

What will it take to make AI work better? .

We need to make sure that we can describe the world as accurately and as faithfully as possible. For example, Facebook has been using computer vision to describe photographs. But in many cases, the only way to do that accurately is to be able to describe not only the objects in the scene, but also the relationships between the objects. We have to teach a machine what is connecting all the objects, what is the meaning of the scene, and what are the implications of the scene for our lives. That’s a big step. That’s going to take a lot longer.