NSA: Hardware‑Optimized Sparse Attention Mechanism from DeepSeek, Peking University and University of Washington

The NSA (Neural Sparse Attention) mechanism, jointly proposed by DeepSeek, Peking University and the University of Washington, addresses performance bottlenecks of traditional attention in long‑context and multi‑turn dialogue scenarios by introducing a hardware‑optimized sparse attention design.

NSA consists of three parallel branches: token compression , which aggregates consecutive key/value blocks into coarse‑grained representations to capture global information; token selection , which ranks compressed blocks and retains the most relevant tokens to reduce computation; and a sliding window that preserves local syntactic and semantic continuity.

A learnable gating mechanism dynamically balances the contributions of the three branches, allowing the model to adaptively weight global versus local attention.

On the hardware side, NSA is implemented with the Triton framework, leveraging group‑shared query loading, KV data stored in high‑bandwidth memory (HBM) and on‑chip SRAM, and optimized memory access patterns to accelerate sparse attention kernels.

The design achieves significant speed‑up while maintaining accuracy, making it suitable for accelerating large language models, long‑document understanding, and other tasks requiring efficient long‑range dependency modeling.

Additional sections provide brief AI market updates (download trends of ChatGPT, Gemini, Perplexity, Claude, etc.) and a list of related DeepSeek technical articles and resources.