Genie: Google DeepMind's Revolutionary Text-to-Video Game Model

Friday - March 1 2024, 14:32 UTC - 8 months ago

Google DeepMind has created a revolutionary model called Genie that can generate playable video games from a short description, hand-drawn sketch, or photo. Unlike previous models, Genie was only trained on video footage, making it more efficient. It generates each frame of the game on the fly based on the player's actions and is capable of learning common visual quirks found in platformer games. In the future, it could potentially generate games at a faster pace and may even be turned into a game-making tool. Genie also showcases the potential for open-ended learning in the AI world.

OpenAI’s recent reveal of its stunning generative model Sora pushed the envelope of what’s possible with text-to-video. Now Google DeepMind brings us text-to-video games.

The new model, called Genie, can take a short description, a hand-drawn sketch, or a photo and turn it into a playable video game in the style of classic 2D platformers like Super Mario Bros. But don’t expect anything fast-paced. The games run at one frame per second, versus the typical 30 to 60 frames per second of most modern games.

As AI researcher Matthew Guzdial from the University of Alberta notes, Genie is a cool piece of work. It was trained on 30,000 hours of video footage from hundreds of 2D platform games found online. Unlike previous models, Genie was trained on video footage alone and does not require input actions for learning.

Genie generates each new frame of the game on the fly depending on the action the player takes. Press Jump, and Genie updates the current image to show the game character jumping; press Left and the image changes to show the character moved to the left. The game ticks along action by action, each new frame generated from scratch as the player plays. But while the current version of Genie runs at one frame per second, future versions could potentially generate games at a faster pace with the use of improved inference speed.

One interesting feature that Genie has learned is the use of parallax, commonly found in platformer games where the foreground moves faster than the background. Genie often adds this effect to the games it generates, making them more visually appealing and realistic.

Although Genie is an in-house research project and will not be released to the public, the Google DeepMind team has hinted at the potential for it to become a game-making tool. Matthew Guzdial himself is also working on a similar project and is excited to see what the Google DeepMind team will create in the future.

But the potential for Genie goes beyond just game generation. The Google DeepMind team also focuses on open-ended learning, where AI-controlled bots are placed into virtual environments and left to solve various tasks through trial and error. In one experiment, Genie was given video footage of small sections of levels and was able to generate new and unique levels that resembled those in the original games.