How Skills Can Cut Costs and Speed Up High‑Quality LLM Data Pipelines

Why High‑Quality Data Is Expensive

When building Retrieval‑Augmented Generation (RAG) knowledge bases, cleaning training data, or generating synthetic data at scale, a single typo in a field name can waste hours of GPU time and incur API charges, turning a promising run into a total loss.

DataFlow‑Skills: Discipline for Intelligent Agents

To address this, the Peking University DCAI team released DataFlow‑Skills, an open‑source set of rules that embed proven DataFlow engineering practices—static verification, recoverable execution, field‑dependency checks, and pipeline planning—into agents. Developers describe desired workflows in natural language; the agent first plans and validates, then executes the costly steps.

Three Core Skills

DataFlow‑Skills currently provides three Skills: generating-dataflow-pipeline – generates a complete, runnable pipeline given a task goal and a JSONL sample. dataflow-dev – a development assistant that creates new operators, writes prompts, diagnoses errors, and reviews code, acting like a senior DataFlow engineer. core_text – an operator‑level API reference containing eight generators, three filters, two refiners, and five evaluators, used by other Skills as structured background knowledge.

Three Design Rules

Rule 1: Reason first, generate later. Each Skill outputs a two‑stage record: a decision log describing chosen operators, field flows, and rationales, followed by the final code. This auditability lets developers catch logical errors before GPU or API budgets are spent.

Rule 2: Prefer specialized operators, use generic ones only as fallback. When a task can be handled by a dedicated operator such as Text2MultiHopQAGenerator, the agent must choose it over a generic PromptedGenerator. Negative constraints (what not to use) provide most of the engineering value.

Rule 3: Treat field dependencies as first‑class citizens. A common bug is a mismatch between an output field (e.g., cleaned_text) and the next operator’s expected input ( clean_text). DataFlow‑Skills embeds field‑dependency chains in the generation rules, preventing references to undefined fields.

Supporting Custom Skills

Teams can add domain‑specific operators by placing SKILL.md and example files under core_text/<category>/<your‑operator>/, then registering the operator in the generating-dataflow-pipeline/SKILL.md table. If the operator becomes widely useful, it can be promoted to a core primitive with updated selection‑priority and signature rules.

DataFlow Architecture

Operator design separates core operators from domain‑specific ones and groups them by behavior (Generate, Evaluation, Filter, Refine). This modularity enables reuse instead of rebuilding prompts for each task.

Generate → Evaluate → Filter → Refine

Syntax constraints enforce field contracts at the init() stage and verify them with compile() before run(). The compile step checks field existence, upstream/downstream key matching, and whether any field is overwritten before definition.

Heterogeneous scheduling maps operators to Ray tasks, allowing explicit CPU/GPU resource declarations via ray.remote. Benchmarks show an 8‑GPU deployment of FlashMineru achieving ~7.6× speedup, and a quality‑filter operator gaining ~6× acceleration when parallelized.

Use Cases

VQA textbook conversion aligns questions, images, and answers across pages, producing multimodal VQA samples ready for training.

Strong reasoning data synthesis starts from seed problems, generates candidate reasoning paths, and validates each step (formula derivation, symbol consistency, computation correctness). After multiple filtering rounds, the pipeline yields dense, consistent reasoning data that matches or exceeds the performance of large open‑source instruction datasets on downstream tasks.

Experiments with only 10 k synthetic samples from DataFlow‑Instruct‑10K achieve near‑official Instruct model performance on mathematics and code benchmarks, while preserving MMLU scores, demonstrating that high‑quality synthetic data can replace much larger instruction corpora.

Conclusion

DataFlow‑Skills shows that cost‑aware pipeline engineering—planning before execution, strict operator selection, and field‑dependency enforcement—can dramatically reduce wasted GPU time and API spend, making high‑quality LLM training data production more reliable and scalable.