Tesla's gamble on developing its own chips

Elon Musk doesn't want Tesla to be just a car maker. He wants Tesla to be an artificial intelligence company, a company that knows how to make cars drive themselves.

Dojo, Tesla’s custom-built supercomputer used to train its Full Self-Driving (FSD) neural network, is critical to the mission. FSD isn’t actually fully autonomous; it can perform some autonomous driving tasks but still requires the driver to pay attention. But Tesla believes that with more data, more computing power, and more training, it can cross the threshold from near-autonomous driving to fully autonomous driving.

Musk has been teasing Dojo, but the executive has been stepping up talk of a supercomputer for 2024. Dojo's importance to Tesla could be existential -- as electric vehicle sales decline, investors want Tesla to go autonomous. Here's a timeline of Dojo mentions and promises.

April 22 – At Tesla’s Autopilot Day, the automaker brought its AI team on stage to talk about Autopilot and Full Self-Driving, and the AI ​​that powers both. The company shared information about Tesla’s custom chips designed specifically for neural networks and self-driving cars.

During the event, Musk revealed that Dojo is a supercomputer for training artificial intelligence. He also pointed out that all Tesla cars produced by then will have all the hardware required for fully autonomous driving and will only require software updates.

February 2 - Musk says Tesla will soon have more than a million connected cars worldwide equipped with the sensors and computing power needed for fully autonomous driving, and touts Dojo's capabilities.

"Our training supercomputer, Dojo, will be able to process large amounts of video training data and efficiently run hyperspace arrays with large numbers of parameters, ample memory, and very high bandwidth between cores. More on that later."

August 14 - Musk reiterates Tesla's plans to develop a neural network training computer called Dojo to "process really large amounts of video data," calling it a "beast." He also says the first version of Dojo will be "about a year away," which would put its release date around August 2021.

December 31 - Elon says Dojo isn't necessary, but it will make autonomous driving better. "Being safer than a human driver isn't enough, Autopilot will eventually need to be 10 times safer than a human driver."

August 19th - Tesla officially announces Dojo at its first AI Day, an event designed to attract engineers to join Tesla's AI team. Tesla also unveiled the D1 chip, which the automaker said it will use (along with Nvidia's GPUs) to power the Dojo supercomputer. Tesla noted that its AI cluster will house 3,000 D1 chips.

October 12 - Tesla released a Dojo technical white paper titled "Tesla Configurable Floating-Point Format and Algorithm Guidelines." The white paper outlines a technical standard for a new type of binary floating-point algorithm for deep learning neural networks that can be implemented "entirely in software, entirely in hardware, or in any combination of software and hardware."

August 12 – Musk says Tesla will “gradually adopt Dojo. Will not need to buy as many incremental GPUs next year.”

September 30th - At Tesla's second AI Day, the company revealed that it had installed the first Dojo cabinet and conducted a 2.2-megawatt load test. Tesla said it builds one tile (composed of 25 D1 chips) per day. Tesla demonstrated Dojo on stage, running a stable diffusion model to create an AI-generated image of "Cybertruck on Mars."

Importantly, the company set a target date for completion of the full Exapod cluster of the first quarter of 2023, and said it plans to build a total of seven Exapods in Palo Alto.

April 19 - Musk tells investors at Tesla's first-quarter earnings conference that Dojo "has the potential to improve the cost of training by an order of magnitude" and "has the potential to be a sellable service that we would offer to other companies the same way that Amazon Web Services offers web services."

Musk also noted that he "sees Dojo as a long shot" but "worth a try."

June 21 - Tesla's AI X account posts that the company's neural networks are already in customer vehicles. The post includes a chart with a timeline of Tesla's current and projected computing power, noting that Dojo will begin production in July 2023, though it's unclear if that refers to the D1 chip or the supercomputer itself. Musk said that day that Dojo is already online and running tasks in Tesla data centers.

The company also predicts that Tesla will be among the top five in the world in terms of computing power by around February 2024 (no indication that this will be successful), and by October 2024, Tesla will have 100 exaflops of computing power.

July 19 - Tesla notes in its second-quarter earnings report that it has begun production of Dojo. Musk also says Tesla plans to invest more than $1 billion in Dojo by 2024.

September 6 – Musk posts on X that Tesla is limited by AI training compute, but that Nvidia and Dojo will fix that. He says it’s extremely difficult to manage the roughly 160 billion frames of video data Tesla gets from its cars every day.

January 24 – During Tesla’s fourth quarter and full year earnings call, Musk again acknowledged that Dojo is a high risk, high reward project. He also said that Tesla is looking at “both Nvidia and Dojo,” that “Dojo is working,” and that “training is working.” He noted that Tesla is scaling up and has “plans for Dojo 1.5, Dojo 2, Dojo 3, and so on.”

January 26 – Tesla announces plans to spend $500 million to build a Dojo supercomputer in Buffalo. Musk subsequently downplays the significance of the investment, posting on X that while $500 million is a large sum, it is "only equivalent to 10k H100 systems from Nvidia. Tesla will invest more in Nvidia hardware this year. The bottom line to be competitive in AI right now is billions of dollars per year at a minimum."

April 30 - According to IEEE Spectrum, at TSMC's North American Technology Symposium, the company said that Dojo's next-generation training module, D2, has been put into production. D2 puts the entire Dojo module on a single silicon wafer, rather than connecting 25 chips to make a module.

May 20 - Musk notes that the rear portion of the Giga Texas factory expansion will include construction of a "super-dense, water-cooled supercomputer cluster."

June 4th - A CNBC report suggests Musk is moving thousands of Nvidia chips reserved for Tesla to X and xAI. After initially saying the report was false, Musk posts on X that Tesla has nowhere to send Nvidia chips to power them up because of the ongoing Giga Texas South expansion, "so they're just sitting in a warehouse." He notes that the expansion will "house 50,000 H100s for FSD training."

He also posted:

“I said Tesla would invest about $10 billion in AI this year, with about half of that being internal spending, primarily Tesla-designed AI inference computers and sensors that are installed in all of our cars, and Dojo. NVidia hardware accounts for about 2/3 of the cost of building an AI training supercluster. My current best guess for what Tesla will acquire from Nvidia this year is $3-4 billion.”

July 1 — Musk reveals at X that current Tesla vehicles may not have the right hardware for the company’s next-generation AI models. He says that “the roughly five-fold increase in the number of parameters for next-generation AI is going to be hard to achieve” without upgrading the car’s inference computer.

July 23 - During Tesla's second-quarter earnings call, Musk said demand for Nvidia hardware is "so high that GPUs are often difficult to get."

"I think as a result we need to invest more effort in Dojo to make sure we have the training capabilities we need," Musk said. "We do see a path to compete with Nvidia with Dojo."

A chart in Tesla’s investor materials predicted that Tesla’s AI training capacity would grow to about 90,000 H100-equivalent GPUs by the end of 2024, up from about 40,000 in June. Later that day, Musk posted on X that Dojo 1 would have “about 8,000 H100-equivalent online training by the end of the year.” He also posted a photo of the supercomputer, which appears to use the same refrigerator-like stainless steel enclosure as Tesla Cybertrucks.

July 30 - Musk says AI5 is about 18 months away from mass production in response to a post claiming to form a club of "Tesla HW4/AI4 owners angry about being behind on AI5 launch".

August 3 – Musk posts on X that he visited the “Tesla supercomputing cluster at Giga Texas (aka Cortex)”. He notes that the cluster will consist of approximately 100,000 H100/H200 Nvidia GPUs and will be equipped with “massive storage for FSD and Optimus video training”.

Today is the 3855th content shared by "Semiconductor Industry Observer" for you, welcome to follow.

★ Important report on EUV lithography machine released by the United States

Silicon carbide "surge": catching up, involution, and substitution

★ Chip giants all want to “kill” engineers!

Apple , playing with advanced packaging

★ A historic moment for GPU!

★ Continental Group, developing 7nm chips

★ Zhang Zhongmou's latest interview: China will find a way to fight back

★ EUV lithography’s new “savior”