MN-Core
3D-Stacked
DRAM
- Stack logic and memory wafers vertically
- Connect compute units and memory across the full wafer area
- Achieve both SRAM-level interconnect density and HBM-level memory capacity
Inference workloads—in which trained AI models process inputs to generate outputs—are evolving rapidly.
Traditional chat has primarily focused on responding to user questions. Reasoning models work through problems step by step during inference, often breaking them into smaller parts, and therefore generate more tokens internally before arriving at an answer. Agent models go further, repeating a cycle of reasoning, tool execution, result evaluation, and action selection as they work toward completing a task.
As thinking processes grow longer and inference is repeated more often, the time spent generating tokens adds up. The speed of each inference step has an increasingly significant impact on the total time needed to complete a task.
The more deeply AI thinks and the more actions AI takes, the more valuable fast inference becomes.
Modern LLMs use Transformer-based architectures. Per-user inference speed is primarily determined by memory bandwidth (the amount of data that can be transferred from memory per unit of time).
Because these models are so large, most of the time required to generate each token is spent loading model weights and the KV cache from memory into the processor.
Despite various architectural innovations aimed at reducing memory traffic, there has been no fundamental change in the three years since ChatGPT’s launch.
Compute performance is now 60,000 times what it was 20 years ago, while memory bandwidth is only 100 times what it was. This persistent shortfall in memory bandwidth has been a major challenge in computing over the past two decades.
To fully utilize a processor’s computing power, data must be supplied fast enough to keep pace. As compute performance increases, so does the demand for memory bandwidth.
In LLM inference in particular, access to model weights and the KV cache has a major impact on performance. Higher memory bandwidth is essential for fast token generation.
Source: Amir Gholami et al., AI and Memory Wall, IEEE Micro, 2024; arXiv:2403.14123
Process scaling has made transistors smaller and reduced the energy required for switching. Interconnects, however, behave differently.
As wires become narrower, the spacing between adjacent wires and the thickness of insulating layers also decrease. Parasitic capacitance per unit length depends on the ratios of wire width to wire spacing and interlayer distance, rather than on wire width alone. Even when all dimensions shrink proportionally, these ratios remain unchanged. As a result, parasitic capacitance per unit length barely decreases as wires shrink. Supply voltage reductions have also largely stalled in recent years, so process scaling does not reduce the energy required to move data over long interconnects.
To reduce the energy consumed by data movement, we therefore need to shorten the distance the data travels.
Data Movement Distance and Energy Consumption per Bit
| Item | 3D-Stacked DRAM | HBM | SRAM | DRAM |
|---|---|---|---|---|
| Structure | ||||
| Data Movement |
< 1 mm | 40 mm | < 1 mm | > 100 mm |
| Energy | ~1 pJ/bit | 4 pJ/bit | ~1 pJ/bit | ~30 pJ/bit |
| Speed | ||||
| Capacity |
MN-Core
Existing Technology A
Existing Technology B
Existing Technology C
Even when memory and logic are stacked, a conventional compute architecture based on shared memory still requires data to travel long distances through multiple cache levels. MN-Core provides each compute unit with its own memory, allowing processing to be completed directly beneath the stacked memory. This enables MN-Core to take full advantage of 3D stacking of logic and memory.
GPU
MN-Core
Thermal management is the greatest technical challenge for 3D-stacked DRAM: AI chips generate substantial heat, while memory is sensitive to temperature. MN-Core’s inherently power-efficient architecture keeps heat generation low, making this approach possible.
See demonstrated power efficiency