HBM saves artificial intelligence

I'm sure we're all familiar with Moore's Law, which to be honest is really more of a casual observation. Back in the era we call 1965, Gordon Moore, co-founder of Fairchild Semiconductor and Intel Corporation (and the former CEO of Intel Corporation), argued that the number of transistors that could be made on a chip was doubling every year. Ichiban. He also predicts that this growth rate will continue for at least another decade.

In 1975, looking back on the past decade and looking ahead to the next, Gordon revised his forecast to a doubling every two years, which is the version most people refer to.

In general, Moore's Law also reflects the amount of computing power provided by microprocessors. Initially, increases in processing power were achieved through a combination of more transistors and higher clock speeds. Later, as clock speeds began to stabilize, the industry moved toward multi-core. Recently, we have begun using innovative architectures and hardware accelerators to extend the multi-core paradigm. And, as for the future, we might reflect on the words of everyone from Nobel Prize-winning quantum physicist Niels Bohr to legendary baseball player (and philosopher) Yogi Berra: “Make predictions It’s very difficult, especially when it comes to predicting the future.”

I couldn't have said it better myself. What I can say is that you can ask me in 10 years and I will tell you what happened.

It is often said that the only two certainties in life are death and taxes. Personally I think we could add "more umph" to this list, where "umph" could be represented by "the exponential growth of the insatiable need for increased processing power, memory size, communication speed*" Such a list would also Continue. (*Once you start saying that, it’s hard to stop.)

That being said, most people give the impression of being relatively satisfied with the processing power of tracking the Moore's Law curve (if you don't use log Y, it's just a curve), at least until things like high-performance computing (HPC) and artificial intelligence (AI) ) and the like appeared.

Back in 2018, the folks at OpenAI pointed out in their Artificial Intelligence & Computing - the company that introduced us to ChatGPT, which ironically now needs no introduction (not even to my 93-year-old mother also know this), the paper believes that artificial intelligence can be divided into two eras. In the first era (from 1956 to 2012), the computational requirements for AI training followed Moore's Law, doubling approximately every two years. In 2012, we reached an inflection point, the second era began, and computing demand began to double every 3.4 months!

Now, you might be saying “Ah, AI, what can you do, huh?” However, as our previous article pointed out – will Intel’s new architectural advancements define the next decade of computing? --At Intel Architecture Day 2021, Raja Koduri (then senior vice president and general manager of the Accelerated Computing Systems and Graphics (AXG) Group) noted that Intel is seeing a need to double processing power (across domains) every 3 to 4 Doubling every month (not just AI).

Now, having a lot of computing power is great, but only if you can feed the processor as much data as possible ("Give me Seymour!").

Sure, we could pack an incredible amount of double data rate synchronous dynamic random access memory (DDR SDRAM, or DDR for short) on the same board as the processor, but we'd still be consuming power and increasing the latency of receiving data from the processor . processor. The current "best in class" solution is to add as much SDRAM as possible within the chip package, which gives us High Bandwidth Memory (HBM).

HBM achieves higher bandwidth than DDR4 or GDDR5 while using less power and a smaller form factor. This is accomplished by creating a stack of DRAM dies, which are typically mounted on top of an optional base die that can include buffer circuitry and test logic. One or more HBM stacks can be mounted directly on top of the main processor chip, or both the HBM and processor can be mounted on a silicon interposer.

I have to admit, the last time I looked at HBM, everyone was very excited about the availability of HBM2E, so you can imagine my surprise and joy when I saw the following "Evolution of HBM Cheat Sheet".

I guess I must have blinked because I now see that we've passed HBM3 and are entering HBM3E territory. "Oh my God," was all I could say. The stack height is 16 dice and the data rate is 9.6 gigabits per second (Gbps). This will no doubt be of interest to people building data center servers for HPC and AI training.

Why does the image only have "HBM3" annotation and not "HBM3E"? The reason is that the image was created before the HBM3E nomenclature was officially adopted as a standard (to be honest, it's probably still just a de facto standard as of this writing, but that's good enough for me).

In addition to delivering a market-leading 9.6Gbps data rate, the latest version of the HBM3 memory controller IP delivers a total interface bandwidth memory throughput of 1,229Gbps, or 1.23 terabytes per second (Tbps).

All I can say is that this will make HPC geeks and AI nerds squeal, like... well, that squeal stuff. What did you say? Could you use this screaming bandwidth in your next design?

*Disclaimer: This article is original by the author. The content of the article is the personal opinion of the author. The reprinting by Semiconductor Industry Watch is only to convey a different point of view. It does not mean that Semiconductor Industry Watch agrees or supports the view. If you have any objections, please contact Semiconductor Industry Watch.

Today is the 3603rd issue of "Semiconductor Industry Observation" shared with you. Welcome to pay attention.