AnyTeleop: AI-Powered Teleoperation For Robotic Arms and Hands

Sunday - August 13 2023, 15:25 UTC - 1 year ago

AnyTeleop is a computer vision-based system created by researchers at NVIDIA and UC San Diego which enables robust and flexible teleoperation of a robotic arm and hand to complete tasks at a distance. It is low-cost, easy to deploy, and generalizes well across different tasks, environments and robotic systems.

Recent advances in the fields of robotics and artificial intelligence (AI) have opened exciting new avenues for teleoperation, the remote control of robots to complete tasks in a distant location. This could, for instance, allow users to visit museums from afar, complete maintenance or technical tasks in spaces that are difficult to access or attend events remotely in more interactive ways.Most existing teleoperation systems are designed to be deployed in specific settings and using a specific robot. This makes them difficult to apply in different real-world environments, greatly limiting their potential.

Researchers at NVIDIA and UC San Diego recently created AnyTeleop, a computer vision–based teleoperation system that could be applied to a wider range of scenarios. AnyTeleop, introduced in a paper pre-published on arXiv, enables the remote operation of various robotic arms and hands to tackle different manual tasks.

"A primary objective at NVIDIA is researching how humans can teach robots to do tasks," Dieter Fox, senior director of robotics research at NVIDIA, head of the NVIDIA Robotics Research Lab, professor at the University of Washington Paul G. Allen School of Computer Science & Engineering and head of the UW Robotics and State Estimation Lab, told Tech Xplore.

"Prior work has focused on how a human will teleoperate, or guide, the robot—but this approach has two barriers. First, training a state-of-the-art model requires many demonstrations. Second, set-ups usually feature a costly apparatus or sensory hardware and are designed only for a particular robot or deployment environment," said Fox.

The key goal of the recent work by Fox and his colleagues was to create a teleoperation system that is low-cost, easy to deploy and generalizes well across different tasks, environments and robotic systems. To train their system, the researchers teleoperated both virtual robots in a simulated environments and real robots in a physical environment, as this reduced the need to purchase and assemble many robots.

"AnyTeleop is a vision-based teleoperation system that allows humans to use their hands to control dexterous robotic hand-arm systems," Fox explained. "The system tracks human hand poses from single or multiple cameras and then retargets them to control the fingers of a multi-fingered robot hand. The wrist point is used to control the robot arm motion with a CUDA-powered motion planner." .

In contrast with most other teleoperation systems introduced in past studies, AnyTeleop can be interfaced with different robot arms, robot hands, camera configurations and different simulated or real-world environments. In addition, it can be applied to both scenarios in which users are nearby and at distant locations.

The AnyTeleop platform can also help to collect human demonstration data (i.e., data representing the movements and actions that humans perform when executing specific manual tasks). This data could in turn be used to better train robots to autonomously complete different tasks.

"The major breakthrough of AnyTeleop is its generalizable and easily deployable design," Fox said. "One potential application is teleoperation-based instruction of robots. An instructor can provide clear guidance and even provide demonstrations in seconds without modifying the set-up or the model." .