AI Imaging Tool Delivers Unprecedented Possibilities for 3D Interactions with 2D Photos

Tuesday - May 30 2023, 10:17 UTC - 1 year ago

A team of computer scientists from the Max Planck Institute for Informatics, MIT, Google and the University of Pennsylvania has developed a new AI imaging tool, called DragGAN, to enable 3D user-interactive editing of 2D images. The new tool can manipulate the colors of an image, create photo-realistic materials and textures, delete elements like buildings or people etc. This AI-based photo editing tool is now available to users everywhere, and its potential to expand and develop is immense.

A team of computer scientists from the Max Planck Institute for Informatics, MIT, Google and the University of Pennsylvania has developed a new AI imaging tool for user-interactive 3D manipulation of 2D images depicted in a photograph. The team published a paper describing the new tool, which is called DragGAN, on the arXiv preprint server along with short videos depicting what the tool can do.

Photoshop was first released back in the late 1980s, and since that time, it and similar apps have been used to edit photographs. Such use has become a standard part of social media—people photoshop images before posting them online as a way to "improve" them. In this new effort, the research team has taken image editing to a whole new level by adding artificial intelligence.

At first glance, DragGAN looks very much like any other image manipulation tool. But videos posted by the creative team clarify that it is capable of doing things no prior application has come even close to achieving, allowing users to alter images in imaginary 3D, on the fly. The researchers call the results "hallucinated occluded content".

Photos, by their very nature, are two-dimensional. Previous photo editing tools have allowed for blurring, coloring or even patching in other imagery. But all such editing is based on user effort—the user has to direct the color correction or blur out wrinkles. An AI-based photo editing tool, taught to recognize features through analyzing thousands or millions of other images, can infer what missing parts of a picture might look like and make changes based on that, with user prompting. In one video, for example, a photograph of an angry person can be changed to show the same person smiling—all with just a click and a drag. The person's face can be turned, as well, revealing parts of the head that were never captured in the original photograph. Likewise, cars, animals or landscapes can be drastically altered using just a few clicks and drags. Adding AI to photo editing adds a whole new dimension to the category—one that could make as big a splash as Photoshop did when it was first introduced.

The paper published by the researchers explains that the new tool can manipulate the colors of an image, create photo-realistic materials and textures, delete elements like buildings or people etc. DragGAN utilizes a Generative Adversial Network (GAN) to generate user-generated 3D editing features with previously unseen accuracy. Furthermore, the application can also be used to generate a sequence of frames to a desired effect, like turning a car into a spaceship. DragGAN has been tested with a range of different datasets, including both real and synthetic data. Moreover, the researchers have provided a way for users to compare the application with other existing image editing tools. In addition, they have shared a dataset consisting of interactive user editing tools that shows the performance of this application with the ability to perform user-interactive editing tasks. Real-time user interaction is possible using the DragGAN application, as the application can be used with only minimum computation delays.

The team believes that this new application will revolutionize the way the industry interacts with digital image data. 2D images can now be manipulated through 3D editing with DragGAN. This AI-based photo editing tool is now available to users everywhere, and its potential to expand and develop is immense. In the future, as AI-based imaging continues to improve, the range of possibilities for image manipulation will only be further extended.