Why More Compute Can't Fix LLM Inference Lag and Why RL Leads to Overtraining

In a deep interview, former Google TPU architect Reiner Pope explains that low‑concurrency fast‑mode services trade higher fees for faster streaming but are limited by memory‑bandwidth bottlenecks, that optimal concurrency balances compute and memory costs, and that pipeline‑parallel sparse expert models and reinforcement‑learning fine‑tuning introduce new inefficiencies and overtraining risks.

LLMMemory BandwidthOvertraining

0 likes · 7 min read

Why More Compute Can't Fix LLM Inference Lag and Why RL Leads to Overtraining

SuanNi

Mar 4, 2026 · Artificial Intelligence

How to Fit Large Language Models into Cars and Robots: A Hardware‑Aware Scaling Law

This article presents a hardware‑aware co‑design framework for edge‑deployed large language models, revealing a scaling law that balances model accuracy and inference latency, and demonstrates how Pareto‑optimal architectures can be discovered quickly using roofline analysis and systematic search on devices like NVIDIA Jetson Orin.

AI inferencePareto optimizationRoofline Model

0 likes · 15 min read

How to Fit Large Language Models into Cars and Robots: A Hardware‑Aware Scaling Law

Baobao Algorithm Notes

Aug 1, 2025 · Artificial Intelligence

Why Training Large Language Models Feels Like Alchemy—and How to Master It

This article breaks down the hardware bottlenecks of large‑scale LLM training, explains the Roofline performance model, arithmetic intensity, and how computation and communication costs interact on GPUs and TPUs, offering concrete formulas and examples for efficient scaling.

Arithmetic intensityDistributed computingGPU

0 likes · 12 min read

Why Training Large Language Models Feels Like Alchemy—and How to Master It