Beyond the GPU: analog, photonic and in-memory compute

If moving data costs far more than computing on it, could a different physical substrate break the deadlock? Two candidates are analog crossbars that compute where the weights are stored, and light that multiplies matrices as it propagates. This lesson explains the physics behind both and the evidence for them as of November 2026, including a long series of NeuroGenesys falsification studies. Most of those studies came back negative or narrowly bounded, and they are reported that way here.

The wall is data movement

In 1995 Wulf and McKee warned that processor speed was outpacing memory speed, so memory would eventually set performance: the "memory wall".^[1] For AI workloads the gap has kept widening. Peak server FLOP/s grew about 3.0× every two years over two decades, against about 1.6× for DRAM bandwidth.^[2] Energy tells the same story. Horowitz's survey of chip energy showed that fetching an operand from off-chip memory costs orders of magnitude more than the arithmetic performed on it.^[3]

For language-model decode this is decisive (see inference economics). On calibrated H100 constants, NeuroGenesys estimated that moving one BF16 weight costs about 40 pJ, while the two FLOPs it enables cost about 1.42 pJ. Making arithmetic entirely free would save only about 3.4% of decode energy.^[4] A proposal that wins only on cheaper multiplication is aimed at the wrong term.

Analog in-memory compute

The physics

Store a weight matrix as conductances $G_{ij}$ in a crossbar of resistive memory devices, such as phase-change memory (PCM), resistive RAM or flash. Apply the input vector as voltages $V_i$ on the rows. By Ohm's law each device passes current $G_{ij}V_i$, and by Kirchhoff's current law each column wire sums them:

That is a full matrix–vector product computed in one step, in place, with no weight fetched from anywhere.^[5]

The silicon

IBM's HERMES chip, fabricated in 14 nm CMOS, integrates 64 analog cores, each a 256×256 PCM crossbar. It reports 16.1 to 63.1 TOPS for 8-bit matrix–vector products, depending on read mode, with near-software accuracy on ResNet and LSTM networks.^[6] A related IBM chip with 35 million PCM devices ran speech-recognition models at comparable accuracy and, by the authors' measure, more than 14× the energy efficiency of conventional hardware.^[7]

The costs sit at the boundary: analog-to-digital converters (ADCs), device drift and noise, and an effective precision of about 4 bits. IBM's Analog Foundation Models work found that hardware-realistic analog noise costs about as much as 4-bit weight-and-input quantization. Distillation-based training on less than 1% of the original pretraining tokens made Phi-3-mini and Llama-3.2-1B degrade much less under that noise.^[8] Attention is harder because its operands are activations, not stored weights. Leroux et al. proposed gain-cell arrays that hold the KV cache itself, so far in simulation at GPT-2 scale.^[9]

What NeuroGenesys's cost models found

Across about 70 pre-registered experiments, the lab's AIMC design study reached a consistent picture. AIMC wins on static-weight matrix multiplies but cannot touch attention's growing KV cache, so its decode-energy advantage over digital dilutes with context length:^[10]

Pure state-space models fail at arbitrary recall, so real models are hybrids. The study's design rule is that the attention window policy decides the outcome, while the attention fraction matters much less. A hybrid with only 8% full-attention layers fell to 1.2–2.3× at 32k or more tokens. A sliding-window hybrid held a constant 5.6–6.8×. On a real RecurrentGemma-2B checkpoint, passkey recall was 1.0 inside the 2,048-token window and 0.0 beyond it.^[10]

Three further limits bound where the approach can go.

• Capacity and area favour digital. Fabricated AIMC stores about 22× fewer weights per mm² than SRAM. Against measured chips, digital won throughput per unit area by about 2.5× at matched granularity. The near-term lane is 1–8B edge models, not 70B datacenter models.^[10][11]
• Prefill is the cleaner niche, but it narrows. The modelled prefill energy win fell from 19.1× at 512 prompt tokens to 1.7× at 32k. It survives only where deployments are capped by power rather than by silicon area.^[11]
• The frontier ceiling is low. For a dense 70B model, the work AIMC replaces is a small share of measured system energy per token, which caps the system-level win at about 1.4× or less.^[10]

The lab's own later audit flagged that its 9–19× prefill and roughly 4× short-context figures are cost-model outputs with no external measurement behind them. It also noted that digital hardware's move to native 4-bit arithmetic has taken over the low-precision territory analog relied on.^[12]

Worked example: Amdahl in watts. If a fraction $f$ of system energy is accelerated by a factor $k$, the overall gain is

The lab measured AIMC at about $k=30.6$ on static-weight matmuls. In its decode-energy model (dense case), the FFN that analog would accelerate was $f=18.8\%$ of energy.^[12] Combining the two, as an illustration:

A 30× component win becomes a 1.2× system win, and an infinite $k$ would give only $1/0.812\approx1.23$.

Photonic compute

A mesh of Mach–Zehnder interferometers can apply a programmable unitary matrix to light, so a matrix–vector product happens in the time light takes to cross the chip. Shen et al. demonstrated a neural-network layer this way in 2017.^[13] In 2025 Lightmatter reported a photonic processor running ResNet, BERT and an Atari reinforcement-learning agent at near-electronic precision on many workloads.^[14]

NeuroGenesys's cost-model study identified four structural walls any optical-compute claim must clear.^[15]

1. Tiling. A mesh smaller than the matrix pays repeated electronic accumulation passes.
2. Precision. Real meshes deliver about 4–5 effective bits.
3. Cascadability. Signal degrades layer by layer and must be regenerated.
4. Nonlinearity. There is no mature cascadable optical nonlinearity.

Its model reproduced a published 2025 coherent chip's throughput exactly (1.28 TOPS). It also showed that the chip's system-level energy, 340 fJ per MAC, was 170× its device-only figure, because converters and lasers dominate. The surviving "fence" was narrow: batch-1, small-width, roughly 5-bit, error-tolerant classification, where optics led by a single-digit factor over a fused GPU, not the 50× headlines.^[15] A follow-up that converted trained networks to all-optical random-feature kernels saw its headline 48× speedup collapse to 1.23× when re-quoted against a digital INT8 ASIC instead of a GPU. Two real advantages survived: robustness to low precision and zero-forgetting frozen features.^[16] Related gates on log-domain arithmetic and photonic "wavefield" graphics closed negative. None of the 16 log-arithmetic ideas was new, and logarithmic number systems lost to standard arithmetic at FP16 and INT8.^[4][17]

Where optics does pay: moving data

Optics has a stronger case for carrying bits than for computing on them. NVIDIA announced co-packaged-optics network switches in 2025, claiming 3.5× better power efficiency than traditional networking methods.^[18] Marvell agreed in December 2025 to acquire Celestial AI, whose optical fabric connects processors to memory, for about $3.25 billion upfront.^[19]

NeuroGenesys's synthesis agrees on direction but narrows the claim. Once laser wall-plug power and realistic link utilization of about 30% were charged, the board-level energy win of optical links disappeared. One configuration came out 3.4× worse than electrical. What survives is bandwidth density at extreme scale, meaning very long contexts, high batch and disaggregated memory, mainly in the decode phase.^[12][20] A brief proposing diamond quantum memory between accelerators failed structurally: AI collectives copy data, which quantum states forbid, and entanglement rates fall about 12.7 orders of magnitude short.^[21]

The baseline keeps moving

Every alternative is measured against a digital target that is improving. Digital designs already capture the "keep the weights still" advantage with on-chip SRAM. Groq's LPU uses SRAM as primary weight storage, and in December 2025 NVIDIA took a non-exclusive license to Groq's inference technology.^[22] Digital formats have also dropped to 4 bits (sparsity and compression). The lab's practical screen, which it calls the Substrate Gate, asks four questions before any hardware build:^[4]

1. Does the advantage shrink as precision drops?
2. Is compute even 5% of the energy?
3. Does the proposal's win double-count with the substrate's?
4. What is the boundary tax in conversions and chained precision?

Key takeaways

• The binding constraint for LLM inference is data movement. Cheaper arithmetic alone barely helps.
• Analog crossbars compute where weights live and work on real silicon at about 4 bits. They cannot accelerate attention's KV cache, so their decode advantage dilutes with context. The design rule that preserves it is sliding-window hybrid architectures.
• Photonic compute is physically real but walled in by tiling, precision, cascadability and conversion costs. Honest baselines shrink its advantages to narrow niches.
• Optics' durable role is interconnect bandwidth density at extreme scale, not compute and not, robustly, energy.
• As of November 2026, no analog or photonic substrate is a general replacement for digital inference hardware.

Sources