What are the challenges of using robots to build art installations?

NVIDIA expert shares the challenges of creating playful and agile robots to build art installations, and her anticipation for the next chapter in autonomous machines.

Training, testing, and coding robots is not easy. Our recently launched Isa platform promises to change that.

No one knows this better than roboticist Madeline Gannon, who has spent the past month tinkering with robotic art installations in her research studio in the Polish Hills outside Pittsburgh.

She is focusing her research efforts on the debut of Manus, a robot that connects 10 arms to a single robotic brain, demonstrating a new frontier in human-robot interaction.

Today, the exhibition is held in Tianjin, China. She has been racing against time and working hard to develop software and interactive design in order to be able to show the flexible and interesting side of these robot arms on time at this exhibition. For several weeks, she has invested 80 hours a week in the warehouse, and the main difficulty she faces is: renting two robots from the robot company ABB, and then using these two robots to interact with all 10 robots to be displayed at the exhibition.

Gannon relies heavily on simulations to create scenes for audience interaction, and she wasn’t sure it would actually work until she had 10 machines up and running on site in China.

Simulating robots in real operations presents challenges that have traditionally driven roboticists to customization (not to mention big gambles and anxiety-inducing events) because, until recently, software couldn’t run stably and reliably.

However, this remains a critical issue in the industry as logistics operations and warehouse operations increasingly rely on robots that have increasing autonomy and are able to work in collaboration with humans.

“As we transition from robotic automation to robotic autonomy, art installations like Manus give people a chance to experience firsthand how humans and autonomous machines will coexist in the future,” she said.

To be sure, Gannon went through a lot of trouble to make this demonstration happen, highlighting the industry’s new dynamic of developing robotics technology on a large scale.

The era of robot assistants has arrived

Most of these are still being refined. Earlier this year, we released Isaac ulat for developing, testing, and training autonomous machines in a virtual environment. Last week at GTC Japan, we announced the Jetson AGX Xavier Devkit to help develop robots and other autonomous machines.

This solution combines software and is driving the robotics revolution by helping to shorten development cycles.

“With Isaac, people are able to develop applications much faster,” Gannon said. “We are in the golden age of robotics right now.”

This isn’t Gannon’s first time competing in robotics competitions. Last year, while a PhD student at Carnegie Mellon University, she developed an interactive arm that was exhibited at the Design Museum in London.

Mimus the robot is a 2,600-pound giant designed to observe its surroundings. The robot is placed in the observation area, using sensors embedded in the museum's ceiling to observe and approach, and even follow, visitors it finds interesting.

Manus, which was exhibited at the World Economic Forum in Tianjin, is the second and more complex robotic device she has developed, as it required creating interactions from scratch using custom software.

Create a flexible and interesting Manus

Building Manus was no easy feat. After arriving in China, Gannon had only 10 days to deploy all 10 robots on site before launching the interactive exhibit. Manus’ robots are arranged in a row on a 9-meter-high pedestal and encased in plexiglass. Twelve depth sensors are mounted on the bottom of the pedestal, allowing the interconnected robots to track and respond to visitor movements.

“This project requires a lot of visual processing, which is why NVIDIA is the brain of it,” Gannon said.

Using this vision system, Manus can move autonomously in response to the people around it: If Manus finds someone interesting, all 10 robot arms will reorient as its robotic gaze follows the target person.

To create the interactive design for Manus, Gannon needed to develop custom protocols and motion solvers for his robot, as well as custom people perception, remote monitoring, and human-robot interaction design software.

She said that until now there has been no reliable technical resources that can work with intelligent robots to complete atypical tasks. Therefore, every time she creates a new robot part, she has to repeat herself and waste time on useless work.

The technical development of the Mamus software stack took up about two-thirds of the project time, leaving only one-third for the core content of the project, namely human-computer interaction design.

Future Robot Training

Implementing robots with Jetson and training them with Isaac Sim will help developers reverse these ratios in similar future projects. And they are well suited for developing and simulating the industrial robots that a large number of companies use for warehouse and logistics operations.

Gannon’s pioneering work in training robots for these obstacles has attracted widespread attention, and over the years she has been known as the “robot whisperer” or “robot tamer.”

She shrugged. "Now, with Isaac, I guess we don't need the robot whisperer anymore."

Editor-in-charge: Peng Jing