RWKV: Next‑Generation Heterogeneous Large Model – Design, Evolution, Performance, and Training Strategies

The talk, originally delivered at DataFunCon 2023, introduces RWKV (Ruakuv) – a next‑generation large model that replaces the quadratic‑complexity self‑attention of Transformers with a linear, state‑based RNN mechanism, achieving O(n) inference complexity.

Key sections include:

Origin and Motivation: RWKV was created to overcome the O(n^2) bottleneck of self‑attention, drawing inspiration from the Attention‑Free Transformer (AFT) and emphasizing a physical‑state analogy.

Design Details: The model uses token‑shift instead of positional encoding, a Channel‑Mix module that mimics multi‑head attention, and a Time‑Mix module that replaces the feed‑forward network, enabling linear‑time inference even for 32K‑token contexts.

Performance Benchmarks: Graphs show RWKV’s inference time and memory usage scale linearly with parameter count, outperforming many Transformer baselines; loss curves demonstrate strong long‑sequence learning up to 8K tokens.

Evolution: From the original version to v4 (introducing channel‑mix and time‑mix) and the latest v5 (larger head size and state), the architecture continuously incorporates the best Transformer capabilities while retaining RNN advantages.

Experimental Results: Zero‑shot evaluations reveal RWKV matches or exceeds Transformer models at 10B parameters; multilingual tests show robust English, Chinese, and Japanese performance; long‑memory examples illustrate accurate recall over thousands of tokens.

Training Pipeline: RWKV is trained on open‑source corpora similar to LLaMA’s Pile, using DeepSpeed for parallelism and supporting modest hardware (e.g., 24 GB GPU for a 0.3 B model, 6 GB RAM for inference of a 7 B model). Fine‑tuning techniques such as LoRA, PEFT, and RLHF are discussed.

Applications: Demonstrations include on‑device generation (mobile phone), role‑playing chat, music generation, and multimodal models; the model can handle prompts up to 100 K tokens after fine‑tuning.

Open‑Source Ecosystem: The core codebase is only ~140 lines, hosted on GitHub (BlinkDL/RWKV‑LM); community tools like RWKV‑Runner enable local deployment with minimal resources, and a leaderboard tracks ongoing progress.

Q&A Highlights: Comparisons with RetNet and LONGNET, discussion of long‑prompt behavior, future directions such as expanding state dimensionality, and plans for scaling to 100 B‑parameter models.

The session concludes with acknowledgments to the speaker Liu Xiao (Co‑founder of Shenzhen Yuanshi Intelligent) and references to further resources, including papers, model repositories, and community Discord channels.