Seeing the World Through New Eyes: Research Unveils a Revolutionary Computer Vision Technology That Can Overcome Obstructions

Thursday - May 11 2023, 07:32 UTC - 1 year ago

A group of researchers from MIT and Rice University recently unveiled a computer vision technology that can enable people to see around corners and beyond obstructions when reflections are captured on a shiny object. This breakthrough has the potential to revolutionize safety and navigation of autonomous vehicles, as well as numerous other fields, including health care and search and rescue.

Imagine seeing around corners or peeking beyond obstacles that once blocked your view. It's like having X-ray vision. It might now be possible thanks to new research.

A group of brilliant researchers from MIT and Rice University unveiled an incredible computer vision technology that can change how we see the world. We could soon live in a world where shiny objects become extraordinary "cameras" that let us view our surroundings through their unique lenses.

Their developed technique operates by capturing images of an object from various angles. Then, converting its surface into a virtual sensor that captures reflections. The artificial intelligence system then maps these reflections, enabling it to estimate depth within the scene and unveil novel perspectives that were once hidden from view.

"We have demonstrated that any surface can be converted into a sensor using this formulation that transforms objects into virtual pixels and sensors. This technique holds enormous potential across multiple domains," explains Kushagra Tiwary, co-lead author of the research and a graduate student in the Camera Culture Group at the Media Lab.

The researchers, including co-lead author Akshat Dave from Rice University, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan, and senior author Ramesh Raskar, are set to present their groundbreaking findings at the prestigious Conference on Computer Vision and Pattern Recognition.

The applications of this breakthrough are virtually limitless, as it empowers users to see around corners and beyond obstructions that traditionally obstructed their line of sight. One area where this technology could be truly transformative is autonomous vehicles.

Imagine a self-driving car utilizing reflections from lamp posts or buildings it passes to understand its surroundings, even when obstructed by a parked truck. It's not just futuristic; it's practical and could revolutionize the safety and navigation of autonomous transportation.

But there are challenges because real life isn't as simple as crime shows, where detectives magically enhance blurry reflections to solve crimes. Reflections can be distorted and offer a limited view of the world. How did the researchers overcome the challenges posed by reflections? They developed a three-step technique called ORCa (Objects as Radiance-Field Cameras).

First, they captured multiple images of an object from different angles, capturing various reflections on its glossy surface. Then, they transformed the object's surface into a virtual sensor that captures light and reflections. And that was it; the shiny object is now a "camera." .

Finally, the system uses the virtual pixels on the object's surface to recreate a 3D environment from the object's perspective.

ORCa's superpowers don't stop there. Analyzing multiple reflections can estimate the depth between the shiny object and other elements in the scene. It even predicts the object's shape. Also, thanks to its 5D radiance field representation of the scene, it can reveal hidden features once concealed by corners or obstructions. It's like seeing the world with a whole new set of eyes! .

The researchers are now exploring applications for this amazing technology in areas such as health care, autonomous vehicles, and search and rescue.