Tesla Launches Cutting-Edge Supercomputer for Self-Driving Car Training Powered by NVIDIA A100 GPUs

To tackle one of the greatest computing challenges ever, Tesla will need unprecedented computing power.

At this week's CVPR (International Conference on Computer Vision and Pattern Recognition), Andrej Karpathy, senior director of AI at automaker Tesla, unveiled the company's internal supercomputer for training deep neural networks for Autopilot and autonomous driving. The cluster uses 8 NVIDIA A100 Tensor Core GPUs with 720 nodes (a total of 5,760 GPUs) to achieve a super-strong performance of 1.8 exaflops.

Karpathy said: "This is an excellent supercomputer. In terms of FLOPS, it is probably ranked fifth among supercomputers in the world."

By putting unprecedented levels of computing power in the automotive industry at the core of its research and development cycle, Tesla is enabling its self-driving car engineers to use cutting-edge technology to do this work efficiently.

NVIDIA A100 GPUs provide acceleration at all scales for the world’s most powerful data centers. Built on the NVIDIA Ampere architecture, the A100 GPU delivers 20 times the performance of the previous generation and can be partitioned into seven GPU instances to dynamically adapt to different needs.

Using (data from) more than one million cars on the road to continuously optimize and iterate new features is Tesla's vertical integration approach to autonomous driving, and GPU clusters are part of it.

From cars to data centers

Tesla’s iterative development process starts with the car, where “shadow mode” silently executes perception and prediction deep neural networks (DNNs) without actually controlling the vehicle.

Any mispredictions and misidentifications are recorded, and these instances are then used by Tesla engineers to create a training dataset containing a variety of complex scenarios to improve the DNN.

Currently, 1 million 10-second clips recorded at 36 frames per second have been collected, with a total data volume of up to 1.5PB. In the data center, Tesla puts the DNN into these scenarios and runs them repeatedly until the DNN can run without errors. Finally, the DNN is sent back to the car and the next cycle begins.

Karpathy said that training DNNs on such large amounts of data in this way requires massive computing power, and Tesla built and deployed the latest generation of supercomputers with built-in high-performance A100 GPUs for this purpose.

Continuous Iteration

In addition to full-scale training, Tesla's supercomputers provide autonomous vehicle engineers with the performance they need to experiment and iterate during the development process.

Karpathy said that the DNN structure currently deployed by Tesla allows a team of 20 engineers to work on a network at the same time, enabling parallel development by isolating different functions.

These DNNs can run through training datasets even faster than before when they were rapidly iterating.

“Computer vision is fundamental to everything we do and is key to making Autopilot possible,” Karpathy said. “To do this, we have to train a huge neural network and do a lot of experiments. That’s why we invest a lot in computing power.”