Syll: Open‑Source Multimodal AI Agent Framework for Secure, Trustworthy Automation

Personal AI agents often face fragmented interfaces, steep teaching curves, opaque execution, and privacy or customization challenges. Most systems rely on APIs or command‑line interfaces, which cannot handle closed‑source desktop software, making low‑barrier teaching and transparent operation difficult.

Unified GUI, CLI, and MCP/API Operation

Syll provides a native multimodal execution capability that simultaneously supports GUI, CLI, and MCP/API interactions. It selects the appropriate execution path based on the task, allowing flexible computer control across visual, batch, and service‑oriented scenarios.

Teach‑Once Skill Generation

Instead of requiring users to write code or define complex rules, Syll records a manual demonstration, captures key visual anchors, mouse/keyboard actions, window states, and contextual information. The recorded demonstration is transformed into a reusable skill that reflects "how you complete the task" rather than static button coordinates.

Transparent, Auditable Execution

Every step of Syll’s execution leaves a traceable record: what it sees, which tools it invokes, waiting points, retries, and channel switches. Users can replay, audit, and retain final control over critical decisions, forming a verification loop that enhances trust for high‑sensitivity deployments.

Local, Modular Architecture for Extensibility

Syll stores memory, skills, rules, and preferences as editable local files, ensuring data privacy and high extensibility. End users can manage models, skills, schedules, and dialogues via a front‑end panel, while developers benefit from a highly modular codebase with clear call chains and independent abstractions, facilitating second‑stage development and custom skill plugins.

The framework is currently in a public alpha stage, with ongoing maintenance and iteration planned to support more real‑world tasks while preserving simplicity and extensibility.