How Legato Gives Robots Legato‑Style Smooth Motion

1. Why Robots Hesitate

When a robot performs tasks such as pouring water, stacking bowls, or folding towels, it often pauses or abruptly switches hands, causing wasted time and possible failure. This hesitation stems from the prevalent Action Chunking technique used in Vision‑Language‑Action (VLA) models, which plans a fixed‑length sequence (e.g., one second) and then executes it.

While chunking improves long‑range coherence and inference efficiency, the transition between consecutive chunks is discontinuous, producing a noticeable “break” similar to splicing two audio recordings. Moreover, because Flow‑Matching VLA models are multimodal, the first chunk may choose a left‑hand grasp while the next chunk may select a right‑hand grasp, leading to a sudden “modal switch” at the chunk boundary.

1.1 Action Chunking: A Double‑Edged Sword

Longer planning horizon yields more coherent motions.

Higher inference efficiency by reducing per‑step model calls.

However, the junction between chunks often exhibits abrupt pauses, jitter, or direction changes, especially in high‑frequency control tasks.

1.2 Real‑Time Chunking (RTC) Patch

RTC attempts to smooth the transition by borrowing the unfinished tail of the previous chunk as a prefix for the next chunk during inference. Although this “relay‑baton” idea improves continuity, it suffers from two fundamental flaws:

Inference‑stage RTC: The model never sees such prefix‑conditioned inputs during training, creating a train‑inference mismatch that can cause “fake multimodal switches.”

Training‑stage RTC: The prefix is hard‑pasted and frozen, so the model still treats the prefix as external scaffolding rather than an integrated part of the motion.

Consequently, continuity remains an externally imposed patch rather than an innate capability.

2. Legato’s Solution: Making Continuity a Model’s Inborn Skill

Legato flips the paradigm: instead of patching continuity at inference, it teaches the model to generate continuous actions directly during training.

2.1 Noise‑Real‑Value Mixing Mechanism

Standard Flow Matching starts from pure noise. Legato introduces a guidance vector ω∈[0,1]^H that mixes noise with the real action at each time step: ω=1: the initial state is the real prefix (fully known). ω=0: the initial state is pure noise (fully unknown). 0<ω<1: a gradual blend creates a smooth transition.

The mixing formula (shown in the paper) combines the real action A and noise ε element‑wise, enabling the model to practice “continue from a partially known state” during training.

2.2 Step‑wise Guided Denoising Dynamics

One‑shot guidance at initialization quickly fades as denoising proceeds, causing the model to forget the prefix. Legato therefore applies the mixing before **every** denoising step, effectively installing a “memory anchor” that repeatedly reminds the model of the prefix throughout the diffusion process.

2.3 Training‑Inference Consistency

Because standard Flow Matching optimizes a loss assuming pure‑noise start, it mismatches Legato’s step‑wise guided dynamics. Legato re‑derives the training objective from the guided dynamics, yielding a new velocity‑field loss that aligns the training speed field with the inference‑time guided speed field (scaled by a guidance strength κ). This eliminates the “dual‑standard” problem.

2.4 Randomized Mixing Parameters

To accommodate varying hardware latencies and task smoothness requirements, Legato randomizes two parameters during training:

d (inference delay): controls prefix length; larger d means longer borrowed prefix for slower hardware.

r (transition length): controls how gradually guidance strength decays; larger r yields smoother transitions, smaller r yields faster response.

Training on a distribution of (d, r) pairs allows a single model to adapt at deployment simply by adjusting these two knobs.

3. Experimental Results

Legato was evaluated on a dual‑arm robot across five representative tasks: stacking bowls, pouring, pick‑and‑place, towel stacking, and drawer opening. These tasks involve both rotational and translational motions and frequent multimodal choices (e.g., left vs. right hand).

3.1 Core Findings

Hesitation Reduction: The pause‑and‑switch behavior observed with RTC disappears; trajectory plots show smooth curves instead of jagged spikes.

Task Completion Time: Average reduction of ~10% across all tasks, with up to >20% improvement on highly continuity‑sensitive tasks such as pouring.

Trajectory Smoothness (NSPARC): Average gain of ~10%, with some tasks exceeding 40% improvement.

Additional ablation studies and simulation analyses are provided in the original paper.

3.2 Deployment Guidance

Empirically, setting d = delay, s = 0.5H, and r = H - d - s (where H is the total sequence length) yields strong performance on most hardware platforms and tasks. Fine‑tuning Legato on top of a well‑trained base Flow Matching model further boosts results.

4. Conclusion

Legato introduces a training‑time continuity mechanism that endows Flow Matching policies with native smoothness, aligns training and inference dynamics, and offers flexible control via randomized mixing parameters. By turning “legato‑style” motion from a post‑hoc patch into an intrinsic capability, Legato advances embodied AI toward reliable real‑world deployment.