Developing Robotic Capabilities with RoboTool

Friday - November 17 2023, 17:19 UTC - 1 year ago

Researchers at Carnegie Mellon University and Google DeepMind co-developed RoboTool, an AI system that can enable robots to use tools more creatively. With this tool, robots can tackle complex real-world scenarios more effectively, explore creative solutions, use natural language, and reason about their environment.

Researchers at Carnegie Mellon University and Google DeepMind recently developed RoboTool, a system that can broaden the capabilities of robots, allowing them to use tools in more creative ways. This system, introduced in a paper published on the arXiv preprint server, could soon bring a new wave of innovation and creativity to the field of robotics. "Tool use is often regarded as the hallmark of advanced intelligence," Mengdi Xu, final-year Ph.D. candidate at Carnegie Mellon University and co-first author of the paper, told Tech Xplore.

"In Wolfgang Koehler's experiments, for instance, apes cleverly stacked crates to access bananas hung out of their reach while crab-eating macaques employed stones as tools to crack open nuts and shells. Beyond using tools for their intended purpose and following established procedures, using tools in creative and unconventional ways provides more flexible solutions but presents far more challenges in cognitive ability." .

Robots often complete manual tasks in standard and repetitive ways without exploring alternative approaches. By exploring more creative ways of doing things, however, they could better tackle complex real-world scenarios. "In robotics, creative tool use is also a crucial yet very demanding capability because it necessitates the all-around ability to predict the outcome of an action, reason what tools to use, and plan how to use them," Peide Huang, co-first author and Ph.D. candidate, said.

The primary objective of the recent work by Xu, Huang, and their colleagues was to devise a system that allows robots to use tools more creatively. Such a tool could help to tackle numerous real-world problems more effectively, for instance, allowing robots to adapt their strategies when trying to grasp objects that are out of reach or to create stepstones to climb to a target location.

"The rise of large language models (LLMs) has tremendously enhanced the functionalities of chatbots, coding automation, and visual content creation," Huang explained. "Beyond these digital interfaces, embodied AI could represent the next frontier in intelligence—one that interacts tangibly with the real world. Robots, serving as the physical extensions of LLMs, present an ideal medium for this exploration." .

The advent of LLMs and their recent rise in popularity encouraged researchers to explore their use in the field of robotics. Past studies demonstrated the potential of these models for improving various robot capabilities, including their communication with users, as well as their reasoning, planning, and task execution.

For instance, Google DeepMind's SayCan tool allows robots to comprehend natural language instructions such as "I spilled my drink, can you help?" and subsequently devise strategies to tackle various domestic chores. Yet, leveraging LLMs to solve problems that require reasoning with implicit constraints set by a robot's body and its surrounding environment remains challenging.

Xu, Huang, and their colleagues set out to explore the use of LLMs to boost the creativity with which robots approach different tasks. In other words, their hope was to create a system that woould allow robots to leverage their toolset in creative and unconventional ways.