Building and Optimizing a Store Smart Assistant with Aily: Architecture, Workflow, and Practical Lessons

1. Introduction

Artificial Intelligence (AI) is booming, and many leading tech companies have launched AI‑based tools. After DeepSeek gained widespread attention during the Spring Festival, the Youzan store technology team began exploring AI‑driven efficiency improvements. This article introduces the store‑level intelligent assistant built on the Feishu Aily platform, covering why Aily was chosen, the benefits achieved, the construction process, and practical tips.

Why we chose Aily

The SaaS system handles a large volume of routine configuration and consultation requests from merchants, leading to four main pain points:

Basic configuration or simple inquiries consume a lot of human resources.

High volume of feedback makes it hard for R&D and support teams to process everything promptly.

Many merchants raise repetitive or related issues that require manual checking.

Documentation is scattered, making knowledge sharing and retention difficult.

To address these issues, we introduced the Aily intelligent assistant, which provides three key capabilities:

AI‑driven answering : Leverages a knowledge base and large language models (LLM) to answer merchant questions quickly and accurately.

Continuous learning : Updates the knowledge base automatically as business evolves.

Deep integration : Connects with internal Youzan systems to fetch data and push results back, eliminating information silos.

Using these capabilities, we built the Youzan Store Business Smart Assistant (hereafter “Store Smart Assistant”).

2. Business Process and Effects

2.1 Business Flow Diagram

The assistant listens to Feishu group messages in real time, uses Aily’s RAG+LLM to provide answers, and offers buttons for further actions (e.g., create a Jira ticket or submit a new request) when the answer is unsatisfactory.

2.2 Deep Integration with Internal Systems

The workflow is tightly coupled with internal services, forming a closed loop from inquiry to issue creation and follow‑up.

2.3 Usage Effects

The assistant significantly improves efficiency for R&D and support engineers. Key metrics after less than two months of operation:

~1,000+ dialogues per month; ~2/3 of topics are filtered out by the assistant before reaching humans.

40+ knowledge items added to the knowledge base via automatic summarization.

300+ issues created, including 10+ urgent issues handled via group chats.

Replication to other business lines takes roughly one person‑day.

3. Construction Process & Tips

3.1 Phased Development

Three stages were defined:

3.1.1 Stage 1 – Basic Q&A Capability

We initially used Aily’s out‑of‑the‑box knowledge‑question‑answer flow. When the knowledge base could not answer, a fallback skill judged whether the request was a problem or a new requirement and automatically created a Jira ticket or demand. Issues discovered:

Some non‑issues were mistakenly turned into Jira tickets.

The model often mis‑classified problems as requirements and vice‑versa.

Result: the assistant increased workload instead of reducing it.

3.1.2 Stage 2 – Re‑engineered Workflow & Knowledge Integration

We shifted from pure knowledge‑Q&A to a workflow‑oriented design, embedding the Q&A as a sub‑step. Improvements included:

Workflow reconstruction : Use skill orchestration to build a complete answer flow.

Empowering submitters : Add buttons for “Submit Issue” or “Create Demand” when the answer is insufficient.

Knowledge base enrichment : Vectorize internal help‑center articles and feed them into the assistant’s knowledge base.

3.1.3 Stage 3 – Optimization & Self‑Learning

Key actions in this stage:

Summarize topic content and sync it to a multi‑dimensional table, which is then imported into Aily’s knowledge base (updates every 3 hours).

One‑click creation of urgent‑issue tickets and automatic group chat invitation with strict permission control.

Replace third‑party LLMs with the company’s own model to reduce cost and improve stability.

3.2 Practical Tips

3.2.1 Topic‑Summarization Skill

When a topic contains the keyword “summary”, the skill traverses all messages, extracts Q&A pairs, uses an LLM to condense them, and writes the result into a multi‑dimensional table that syncs back to the knowledge base.

Data flow:

Detect “summary” keyword → trigger skill.

Iterate over all messages in the topic.

Extract Q&A, summarize with LLM.

Write to multi‑dimensional table → sync to knowledge base (auto‑incremental every 3 h).

3.2.2 Modular Skill Design

Encapsulate reusable logic (e.g., token generation, message card creation, summarization) as independent skills, then invoke them from a main workflow to improve maintainability.

Examples of encapsulated skills:

Token acquisition method.

Various message‑card generators.

Topic‑summarization logic.

… (see diagram)

3.2.3 Reducing Hallucinations

Hallucinations arise from vague knowledge. Mitigation strategies:

Refine prompts with explicit constraints.

Lower temperature (0.3‑0.5) to increase determinism.

Enrich the knowledge base with accurate Q&A pairs, terminology, and feedback loops.

Add source citations and disclaimer “AI‑generated answer, for reference only”.

Before optimization the model produced a non‑existent feature (see image). After applying the above measures it correctly identified the limitation.

3.2.4 Model‑Call Timeout Handling

Initially all LLM nodes used gpt-4o . After frequent timeouts, we switched:

Image parsing → gpt-4o

Summarization / intent detection → deepseek-v3

Field extraction → doubao大模型

Reducing the number of LLM calls in the workflow also improves stability.

3.2.5 Managing Context Length

Long prompts exceed model context windows, causing loss of earlier information. Optimize prompts to keep token count within limits.

Example: placeholders ${var1} and ${var2} inflated the prompt to >10 k tokens, leading to incorrect answers. After trimming to ~3 k tokens the assistant responded correctly.

3.2.6 Skill Versioning

Skill orchestration does not support automatic rollback. Before publishing changes, duplicate the skill, test the copy, and only switch to the new version after verification. This preserves a fallback in case of issues.

3.2.7 Use Evaluation Feature Before Release

Because LLM answers are probabilistic, run a batch of test cases through Aily’s “evaluation” feature after any model or prompt change to ensure expected behavior.

4. Summary & Outlook

The development of the Aily intelligent assistant demonstrates that true AI transformation is not about replacing humans but about human‑machine collaboration. By leveraging Aily’s deep integration, we have turned a simple Q&A bot into a comprehensive, low‑code AI platform that automates issue handling, knowledge management, and urgent‑response workflows.

Beyond the assistant, we have applied the same approach to:

AI‑assisted code review practice

Automated online log‑alert analysis using Aily.

With the AI + low‑code capabilities of Aily, we can rapidly prototype and launch new AI applications, accelerating innovation and iteration. As the assistant now handles ~80 % of routine queries, engineers can focus on higher‑value, creative work—fulfilling the ultimate purpose of AI: amplifying human potential.