At AMD's Financial Analyst Day earlier this month (which was actually more interesting than it initially sounds), AMD finally confirmed that it was looking to use high-bandwidth memory (HBM) in an upcoming high-end GPU product. Unfortunately, the company gave away few specifics, other than that HBM uses a form of 3D stacked memory, and that it'll (of course) vastly increase performance while still reducing power consumption.

Stacked memory itself isn't an entirely new technology, but AMD's implementation—which gives its GPUs access to much more memory bandwidth—is a big step forward for a graphics card market that's rapidly approaching the limits of GDDR5. With Nvidia also looking to incorporate a form of HBM in its 2016 Pascal architecture, you're going to be hearing a lot more about this new memory technology over the coming year.

Why do we need HBM?

A suitable replacement for the hard-working, but ageing synchronous dynamic random-access memory (SDRAM) standard has been a long time coming. While the current DDR3 memory standards—as well as offshoots like GDDR5—have been serving the CPU and GPU well, they're starting to show signs of being based on early-'90s technology. Essentially, each revision of SDRAM makes use of the same double data rate (DDR) principle as the original technology, which syncs memory to a system bus (allowing it to queue up one process while waiting for another), and also transfers data on both the rise and fall of the clock signal in order to work twice as fast.

DDR2 further refined this idea by running its internal clock at half the speed of the data bus. This trick not only allowed it to produce a total of four data transfers per internal clock cycle (effectively running twice as fast as DDR), but the slower clock speed also reduced the voltage requirement to 1.8v. DDR3 halved the internal clock again, resulting in a quadrupled clock signal for even faster performance and increased transfer rates of up to 17GB/sec per module, and 1.5v of operating voltage. The latest all singing, all dancing DDR4 memory (currently limited to the X99 chipset) makes use of a new bus, higher clock speeds, and denser chips in order to reach its maximum transfer rate of 25.6GB/sec per module, and a lower 1.2v of operating voltage.

DDR4's lower power consumption and higher density will benefit mobile devices and server farms, but from a performance point of view, there's not much to write home about. While there's long been talk of hitting the so-called "memory wall" (where the execution bandwidth of a CPU is limited by memory bandwidth) at around 16 CPU cores, that's not currently much of an issue for the desktop, where a lower latency is currently preferred over a high-bandwidth solution. In the GPU world, though, bandwidth is king, which is why you often see another type of SDRAM in graphics cards.

On graphics cards, SDRAM is known as GDDR5, which is an offshoot of DDR3 optimised for graphics use. The aim with graphics memory is to make it as wide and as fast as possible to cope with massive chunks of data like textures and data sets for compute work moving in and out of it, especially when working at higher resolutions like 4K where large textures are common. Compared to the 64-bit-wide interface of standard DDR3 memory, GDDR5 goes as wide as a 512-bit interface. Because GPUs are highly parallelised, latency is far less of an issue, so GDDR5 trades higher latency for higher bandwidth. As an example, AMD's R9 290X graphics card sports a total memory bandwidth of up to 352GB per second. Even the fastest DDR4 memory running in a quad-channel configuration would only hit just over 100GB/sec.

The trouble with GDDR5 is that the wide buses it uses require more chips and take up more physical space, which isn't ideal when graphics cards are already rather large. Increasing power consumption isn't an option either, particularly when cooling a GPU like the R9 290X with its 300-watt TDP already presents challenges. Both of these issues also spell trouble for the mobile world, small form factor desktops, and servers, where space and power consumption (as well as cooling options) come at a premium.

As AMD's chart above illustrates, the company believes GDDR5 is about to hit the point where it pulls so much power away from the GPU, overall performance drops. To put it another way, let's say you have a fixed amount of power consumption that you can effectively keep cool, about 200 watts or so. If GDDR5 uses 50 watts of that power, you're left with 150 watts for the GPU. If you could dramatically reduce the power consumption of the memory to 10 watts, while maintaining the same level of performance, you could then pump 40 watts back into the GPU and gain more compute.

The solution, then, to better desktop performance, and something more mobile-friendly, is a memory system that takes up less physical space, has a higher bandwidth, and uses far less power. Or, as AMD puts it, a technology that ups the bandwidth-per-watt and the bandwidth-per-square-millimetre of die space. This is what the company thinks it's cracked with HBM.

How does HBM work?

One of the solutions to increasing the speed and/or bandwidth of a particular component is to integrate it onto the CPU or GPU die. Floating point units, northbridge units, and even GPUs have all found their way directly onto CPUs over the years. Things are a little tricky with DRAM. Traditionally, memory integrated onto a CPU die comes in the form of static random-access memory (SRAM), which is used for things like internal caches. The trouble with SRAM is that, while it's extremely high-bandwidth, it's also extremely expensive (it takes up a large amount of die space). Caches tend to be measured in the low kilobytes or megabytes, rather than the multiple gigabytes seen on graphics cards.

Intel's had a bash at integrating DRAM onto its CPUs in the form of on-package (not on-die) eDRAM, which is part of the company's Crystalwell Iris Pro graphics. It's also used in Microsoft's Xbox 360 and Nintendo Wii U to supplement system memory. However, eDRAM is still an expensive proposition, and once again, there are significant space limitations on the amount of memory that can be integrated. Still, moving the memory closer to the CPU or GPU does dramatically increase the bandwidth and performance, and that's the mechanic that HBM exploits.

HBM is a new, stacked memory design manufactured by Hynix (with other companies jumping on board later) that brings the DRAM as close as possible to the logic die. In a traditional setup, the individual DRAM chips are placed side by side, and connected to the GPU via long copper traces on a PCB. HBM, on the other hand, stacks the memory chips on top of each other, dramatically reducing the overall footprint required, allowing for the use of extremely wide data buses and much slower clock speeds in order to hit the required levels of performance. With HBM, instead of buses being hundreds of bits wide, they can be thousands of bits wide, and because everything is so close, the power consumption per transported bit is much lower.

Stacked chip designs have been around for a little while, but AMD's implementation makes use of two key pieces of technology in order to connect the memory together and get it to talk to the GPU: through silicon vias (TSVs), and a silicon interposer. As shown in the diagram above, the silicon interposer is essentially a small circuit board that sits on top of the entire GPU package substrate and routes signals from the simpler logic of the memory through to the more complex logic of the GPU. With the stacks of memory now sat directly beside the GPU, the traces to connect them are far shorter, enabling the use of much wider buses.

Connecting the stack of DRAM dies to the interposer are TSVs. This is a fine pitch interconnect that's poked directly through the stack of DRAM all the way down to logic die (the controller). In AMD's implementation, the logic is actually distributed across the DRAM stack and a logic die, which is then connected to the interposer with more TSVs. The company notes that, because of the use of a TSV, the memory die has to be very thin—much thinner than usual. The entire stack—the interposer, the logic die, the four DRAM dies—comes in at around 100 microns thick, or as thin as a piece of paper, which is an impressive feat of engineering no matter how you slice it.

Listing image by Flickr