Nvidia's Senior Research Manager and Lead of the Embodied AI division Generalist Embodied Agent Research (GEAR) Lab, Jim Fan, highlighted how artificial intelligence (AI) robots in the future could be trained and function. The executive claimed that soon embodied AI agents will be born in simulation where they will learn and gain expertise in specific tasks. He also highlighted that soon entire cities, houses, and factories will be transported to simulations, making training AI on real-world scenarios possible. He also postulated that future AI robots could share a hive mind.
Nvidia's Jim Fan Shares How AI Robots Could Be Trained in the Future
Fan shared his ideas in a post on X (formerly known as Twitter), as he highlighted that the City of Tokyo has released a 3D digital twin of the entire city in a high-resolution point cloud, and made it free to download. Sharing the development, the Nvidia executive stated that this trend of taking real-world institutions to digital simulations is only going to grow in the future.
“It's an inevitable trend that more and more cities, houses, and factories will be transported into simulations,” Fan said.
Sharing the example, Fan also added that in the future, robots will not be trained in isolation. Currently, robots are trained in controlled environments where they can learn how to move around, identify objects, pick and drop things, and complete tasks. However, many have claimed that this method of training does not prepare the robots for various challenges they might face in real-world scenarios.
Offering a solution to the problem, the Nvidia research manager claimed that soon embodied AI agents will be simulated as an iron fleet. Notably, embodied AI agents are AI systems integrated with a physical body or a simulated embodiment, enabling them to interact with and perceive the world like humans or animals.
Fan said that future AI robots will be deployed in real-time graphics engines and will be scaled across a huge cluster. Such training methods will also produce trillions of high-quality training tokens, he added. Taking the hypothetical vision further, he said, “The majority of embodied agents will be born in sim, and transferred zero-shot to our real world when they are ready.”
Once these AI robots have been deployed in the real world, such as factories or as assistants in offices and homes, they will also share a hive mind. Such hive minds can enable the embodied agents to learn from thousands of use cases as well as coordinate difficult tasks with multi-agent efforts.
While this might appear to be science fiction, Fan, who is leading the embodied AI division at Nvidia, is confident that this will be the direction to train highly efficient robots in the near future. And it appears that Nvidia is already moving in that direction. He shared that the company's Santa Clara headquarters buildings are “designed and rendered in Omniverse, a GPU-accelerated graphics platform, before materialising in atoms.”
