Learning Long-Horizon Surgical Robot Tasks via Transition State Clustering, SWIRL, and DDCO

Recent advances in deep imitation learning and deep reinforcement learning have shown great promise for learning robot control policies from high‑dimensional sensor inputs. However, scaling these methods to long‑horizon sequential tasks requires an impractically large amount of demonstration data because of the credit‑assignment problem.

This article reviews three recent papers that address the problem by decomposing long tasks into shorter sub‑tasks. The proposed algorithms are Transition State Clustering (TSC), Sequential Windowed Inverse Reinforcement Learning (SWIRL), and Deep Discovery of Continuous Options (DDCO). All three can be viewed as special cases of a unified hierarchical framework in which demonstrations are modeled as a sequence of unknown closed‑loop policies that switch at learned transition states.

Transition State Clustering (TSC) identifies robust transition events across demonstrations by clustering candidate transition points in a Gaussian Mixture Model (GMM). The method first segments each trajectory, then clusters the segment endpoints, handling heterogeneous scales of kinematic and visual features with a hierarchical GMM.

Sequential Windowed Inverse Reinforcement Learning (SWIRL) uses the transition states discovered by TSC to define a sequence of local quadratic reward functions. By augmenting the state space with binary indicators of whether a transition region has been reached, SWIRL applies a variant of Q‑learning to learn a policy that respects the ordered reward structure.

Deep Discovery of Continuous Options (DDCO) extends the framework to learn parametrized options (temporally extended actions) directly from demonstrations. DDCO maximizes the likelihood of demonstration trajectories using an expectation‑maximization algorithm that jointly infers option boundaries, termination probabilities, and option policies.

The authors evaluate these methods on several surgical‑robot tasks using the da Vinci platform. In a pattern‑cutting task, a deterministic finite automaton (DFA) with ten primitive actions and two vision‑based checks was manually designed, and TSC automatically recovered the transition structure, discovering an extra transition not annotated by humans.

In a deformable‑sheet tension task, SWIRL identified four meaningful segments (approach, grasp, lift, and flatten) and achieved a four‑fold reward improvement over pure behavior cloning.

In a needle‑picking and placing task, DDCO learned four options that correspond to distinct visual, grasp, and kinematic features. The learned policy succeeded in 7 out of 10 trials, with a 66 % raw grasp success rate that rose to 97 % after filtering out grasp errors.

Overall, the work demonstrates that learning hierarchical task structures from demonstrations can dramatically improve the efficiency and performance of robot learning in long‑horizon surgical applications, and suggests future directions such as better geometric representations, hybrid primitive models, and combined time‑ and state‑space segmentation.