Beyond Orchestrating Workflows: How UnityMAS-O Trains LLM-Based Multi‑Agent Systems

From Workflow Orchestration to Trainable Process

UnityMAS‑O’s basic idea is that users first define a multi‑agent workflow; the framework then transforms this workflow into a reinforcement‑learning problem that can be optimized.

Core Abstractions

The system revolves around four objects: logical roles (planner, retriever, coder, reflector, etc.), the workflow graph, the mapping from roles to physical LLM models, and a reward‑allocation mechanism based on multi‑agent trajectories.

Training therefore evaluates the entire collaborative process rather than a single model’s answer.

Graph‑Structured Trajectories

Instead of collapsing execution to input‑output pairs, UnityMAS‑O records the full execution as a graph‑structured trajectory, preserving retrieval results, tool calls, intermediate states, feedback, and reflections, allowing rewards to be assigned to specific roles and steps.

Decoupling Logical Roles from Physical Models

Logical roles are independent of the underlying LLM parameters; a single model can serve multiple roles or each role can use a distinct model. This enables three sharing schemes: full sharing, full independence, and partial sharing, balancing resource cost, specialization, and training stability.

System Implementation

The architecture combines a central controller with local worker groups. The controller executes the workflow, schedules roles, calls tools, computes and aggregates rewards, and maintains global state. Workers generate rollouts, compute advantages, and perform PPO updates.

This separation lets workflow execution and model updates operate independently, and role‑to‑model mappings can be reconfigured without rewriting the pipeline.

Experimental Setup

Two task families were evaluated. QA and agentic search used Natural Questions and HotpotQA; the code‑generation task employed a three‑step Plan→Code→Verify→Reflect loop with test‑case execution as feedback.

QA and Search Results

After multi‑agent RL training, all workflows showed higher validation‑set F1 scores, especially for smaller models. For a 0.5 B model, previously unstable workflows became practically usable after training.

Parameter Sharing vs. Independent Parameters

On the HotpotQA M‑ASK workflow, the independent‑parameter setting converged slightly faster, but the shared‑parameter setting reached comparable validation F1, indicating that full sharing can reduce resource usage without large performance loss.

Code Generation Task

The reward is the all‑passed test rate. Training increased the rate of 3xQwen3‑4B from 0.255 to 0.686 and 3xQwen3‑8B from 0.290 to 0.738, showing that both the final code quality and the collaborative planning, coding, and reflection improve.

Fewer Verification Rounds

As training progressed, the average number of verification rounds per code task decreased, indicating that the system resolves tasks with fewer reflection steps.

Conclusion

UnityMAS‑O demonstrates that LLM‑based multi‑agent systems can be trained as unified workflows, supporting various sharing schemes and enabling credit assignment across roles. While it does not solve all multi‑agent training challenges, it points toward a future where workflows themselves become optimization targets.