Simultaneous Speech Translation: Technical Background, System Architecture, and Key Challenges

The article introduces simultaneous translation (speech‑to‑text) as a real‑time task that converts source‑language audio into target‑language text, contrasting it with offline speech translation where the full audio context is available.

Two main technical routes are described: end‑to‑end models that directly map audio to text, and cascaded systems comprising Automatic Speech Recognition (ASR) followed by Machine Translation (MT). Industrial practice favors cascaded approaches due to limited bilingual data for end‑to‑end training.

ASR Technology : Modern ASR pipelines consist of audio feature extraction (Transformer or Conformer) and text generation (CTC or AED). Research focuses on streaming decoding, including Transducer models (RNN‑T, Transformer‑T), chunk‑wise attention, and incremental decoding. Strategies such as ensemble de‑noising, context‑aware re‑ranking, and domain‑controlled training improve accuracy and consistency.

MT Technology : Transformer‑based models dominate, with techniques like Deep Transformer (layer‑norm pre‑placement) to increase depth without excessive computation, and data‑augmentation methods such as back‑translation. Non‑autoregressive decoding offers speed gains but requires iterative refinement or teacher‑model distillation to maintain quality.

Challenges include single‑point technical issues (ASR accuracy, domain adaptation, style transfer), system‑level error amplification in cascaded pipelines, and real‑time constraints of simultaneous translation (balancing latency and quality, handling jump‑back corrections).

System Architecture : The presented architecture has two branches—speech‑to‑speech and speech‑to‑text—centered on ASR and MT services, supplemented by a Text Editor (TE) module for post‑processing (disfluency removal, correction, punctuation). Both large‑scale pre‑training and domain‑specific fine‑tuning are employed, using MindSpore and Huawei AI chips for distributed training.

Domain adaptation strategies include large‑model pre‑training, adapter‑based fine‑tuning, LoRA, SpecAugment, and R‑Drop for low‑resource scenarios. Multi‑task learning (ASR, translation, language detection) and multilingual training further enhance robustness.

For streaming scenarios, the system uses CTC for fast initial decoding (process state) and AED for final refined output (final state), with knowledge‑distillation to mitigate inconsistencies.

Additional engineering optimizations involve inference acceleration (CPU quantization, Bolt framework), length‑controlled translation for UI constraints, and keyword‑level evaluation (F1) beyond BLEU.

The solution is deployed internally at Huawei across various products and services, leveraging multimodal translation capabilities on Huawei Cloud.