01 / 07
ENJP
01 — Fast Inference

Fast inference matters
more than ever

Inference workloads—in which trained AI models process inputs to generate outputs—are evolving rapidly.

Traditional chat has primarily focused on responding to user questions. Reasoning models work through problems step by step during inference, often breaking them into smaller parts, and therefore generate more tokens internally before arriving at an answer. Agent models go further, repeating a cycle of reasoning, tool execution, result evaluation, and action selection as they work toward completing a task.

As thinking processes grow longer and inference is repeated more often, the time spent generating tokens adds up. The speed of each inference step has an increasingly significant impact on the total time needed to complete a task.

The more deeply AI thinks and the more actions AI takes, the more valuable fast inference becomes.

02 — Transformer

Memory bandwidth is
all Transformers need.

Modern LLMs use Transformer-based architectures. Per-user inference speed is primarily determined by memory bandwidth (the amount of data that can be transferred from memory per unit of time).

Because these models are so large, most of the time required to generate each token is spent loading model weights and the KV cache from memory into the processor.

Despite various architectural innovations aimed at reducing memory traffic, there has been no fundamental change in the three years since ChatGPT’s launch.

03 — Memory Wall

Compute raced.
Memory did not.

Compute performance is now 60,000 times what it was 20 years ago, while memory bandwidth is only 100 times what it was. This persistent shortfall in memory bandwidth has been a major challenge in computing over the past two decades.

To fully utilize a processor’s computing power, data must be supplied fast enough to keep pace. As compute performance increases, so does the demand for memory bandwidth.

In LLM inference in particular, access to model weights and the KV cache has a major impact on performance. Higher memory bandwidth is essential for fast token generation.

Source: Amir Gholami et al., AI and Memory Wall, IEEE Micro, 2024; arXiv:2403.14123

04 — Data Movement

Bandwidth is a battle
of distance.

Process scaling has made transistors smaller and reduced the energy required for switching. Interconnects, however, behave differently.

As wires become narrower, the spacing between adjacent wires and the thickness of insulating layers also decrease. Parasitic capacitance per unit length depends on the ratios of wire width to wire spacing and interlayer distance, rather than on wire width alone. Even when all dimensions shrink proportionally, these ratios remain unchanged. As a result, parasitic capacitance per unit length barely decreases as wires shrink. Supply voltage reductions have also largely stalled in recent years, so process scaling does not reduce the energy required to move data over long interconnects.

To reduce the energy consumed by data movement, we therefore need to shorten the distance the data travels.

Data Movement Distance and Energy Consumption per Bit

05 — Memory Technologies

3D-Stacked DRAM enables
ultimate memory bandwidth.

Item 3D-Stacked DRAM HBM SRAM DRAM
Structure
Data
Movement
< 1 mm40 mm< 1 mm> 100 mm
Energy ~1 pJ/bit4 pJ/bit~1 pJ/bit~30 pJ/bit
Speed
Capacity

MN-Core

3D-Stacked
DRAM

  • Stack logic and memory wafers vertically
  • Connect compute units and memory across the full wafer area
  • Achieve both SRAM-level interconnect density and HBM-level memory capacity

Existing Technology A

HBM

  • Logic and DRAM are mounted side by side
  • Slower than SRAM
  • Memory and logic cannot be placed any closer, so memory bandwidth hits its limit

Existing Technology B

SRAM

  • Memory cells are embedded directly in the logic wafer
  • Extremely fast
  • Low density and limited capacity: even running an LLM with a few billions of parameters requires a rack-scale system

Existing Technology C

DRAM

  • Mounted on the circuit board away from the logic chip and connected by traces over 100 mm long
  • High capacity, but requires high energy at approximately 30 pJ per bit
  • Bandwidth is limited by pin counts and trace length
06 — MN-Core Architecture

Optimal for
3D-Stacked DRAM.

Even when memory and logic are stacked, a conventional compute architecture based on shared memory still requires data to travel long distances through multiple cache levels. MN-Core provides each compute unit with its own memory, allowing processing to be completed directly beneath the stacked memory. This enables MN-Core to take full advantage of 3D stacking of logic and memory.

GPU

Shared Memory
Architecture

MN-Core

Fully Distributed
Memory Architecture

07 — Efficiency

World's most
efficient architecture

An efficient architecture that enables 3D stacking

Thermal management is the greatest technical challenge for 3D-stacked DRAM: AI chips generate substantial heat, while memory is sensitive to temperature. MN-Core’s inherently power-efficient architecture keeps heat generation low, making this approach possible.

See demonstrated power efficiency