Automated Test Case Generation Using LangChain, Vector Databases, and Large Language Models

In the context of agile team building, the author explored automated unit testing with a Suite executor and then investigated additional runners, focusing on using large language models (LLMs) to automatically generate comprehensive test cases.

The existing tool JoyCoder requires multiple manual steps (copy‑paste, prompt crafting, result copying, saving) and suffers from long response times and token‑limit errors when processing large requirement or design documents.

A comparison between JoyCoder and a self‑developed LangChain‑based solution shows that the latter reduces generation time from 10‑20 minutes (with many manual interactions) to about 5 minutes for large documents, while maintaining comparable accuracy and offering easier prompt optimization.

LangChain is an open‑source framework for building LLM‑driven applications; it supports various document loaders (including PDF) and provides components such as prompt templates, memory modules, and integrations with vector stores.

Three generation strategies are described: (1) feed the full product requirement and design documents to the LLM; (2) feed a summarized version of the documents to avoid token limits; (3) store the documents in a vector database, retrieve relevant chunks, and generate test cases for specific sections.

Implementation details include using PyMuPDF for PDF parsing, LangChain’s text splitters to chunk large files, ConversationBufferMemory and ConversationSummaryBufferMemory to retain context, and Vearch as the vector database (IVFFLAT index) for similarity search.

Code framework and key snippets:

def case_gen(prd_file_path, tdd_file_path, input_prompt, case_name):
    """用例生成的方法
    参数:
    prd_file_path - prd文档路径
    tdd_file_path - 技术设计文档路径
    case_name - 待生成的测试用例名称
    """
    # 解析需求、设计相关文档, 输出的是document列表
    prd_file = PDFParse(prd_file_path).load_pymupdf_split()
    tdd_file = PDFParse(tdd_file_path).load_pymupdf_split()
    empty_case = FilePath.read_file(FilePath.empty_case)

    # 将需求、设计相关文档设置给memory作为llm的记忆信息
    prompt = ChatPromptTemplate.from_messages([
        SystemMessage(content="You are a chatbot having a conversation with a human."),
        MessagesPlaceholder(variable_name="chat_history"),
        HumanMessagePromptTemplate.from_template("{human_input}")
    ])
    memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
    for prd in prd_file:
        memory.save_context({"input": prd.page_content}, {"output": "这是一段需求文档，后续输出测试用例需要"})
    for tdd in tdd_file:
        memory.save_context({"input": tdd.page_content}, {"output": "这是一段技术设计文档，后续输出测试用例需要"})

    # 调用大模型生成测试用例
    llm = LLMFactory.get_openai_factory().get_chat_llm()
    human_input = "作为软件测试开发专家，请根据以上的产品需求及技术设计信息，" + input_prompt + ",以markdown格式输出测试用例，用例模版是" + empty_case
    chain = LLMChain(llm=llm, prompt=prompt, verbose=True, memory=memory)
    output_raw = chain.invoke({'human_input': human_input})

    # 保存输出的用例内容，markdown格式
    file_path = FilePath.out_file + case_name + ".md"
    with open(file_path, 'w') as file:
        file.write(output_raw.get('text'))

def case_gen_by_vector(prd_file_path, tdd_file_path, input_prompt, table_name, case_name):
    """当文本超级大时，防止token不够，通过向量数据库，搜出某一部分的内容，生成局部的测试用例，细节更准确一些"""
    # 解析需求、设计相关文档, 输出的是document列表
    prd_file = PDFParse(prd_file_path).load_pymupdf_split()
    tdd_file = PDFParse(tdd_file_path).load_pymupdf_split()
    empty_case = FilePath.read_file(FilePath.empty_case)
    # 把文档存入向量数据库
    docs = prd_file + tdd_file
    embedding_model = LLMFactory.get_openai_factory().get_embedding()
    router_url = ConfigParse(FilePath.config_file_path).get_vearch_router_server()
    vearch_cluster = Vearch.from_documents(
        docs,
        embedding_model,
        path_or_url=router_url,
        db_name="y_test_qa",
        table_name=table_name,
        flag=1,
    )
    # 从向量数据库搜索相关内容
    docs = vearch_cluster.similarity_search(query=input_prompt, k=1)
    content = docs[0].page_content

    # 使用向量查询的相关信息给大模型生成用例
    prompt_template = "作为软件测试开发专家，请根据产品需求技术设计中{input_prompt}的相关信息:{content},以markdown格式输出测试用例，用例模版是:{empty_case}"
    prompt = PromptTemplate(input_variables=["input_prompt", "content", "empty_case"], template=prompt_template)
    llm = LLMFactory.get_openai_factory().get_chat_llm()
    chain = LLMChain(llm=llm, prompt=prompt, verbose=True)
    output_raw = chain.invoke({'input_prompt': input_prompt, 'content': content, 'empty_case': empty_case})
    # 保存输出的用例内容，markdown格式
    file_path = FilePath.out_file + case_name + ".md"
    with open(file_path, 'w') as file:
        file.write(output_raw.get('text'))

Experimental results on a small project (2363‑word PRD and 158‑word design doc) show a roughly 50% reduction in test‑case design time, with advantages such as rapid comprehensive coverage, reduced manual effort, richer test content, and focused generation via vector search; disadvantages include limited understanding of complex flowcharts and potential mismatch with experienced testers' habits.

Future work includes improving OCR extraction for diagram‑based PDFs, extending LLM assistance to diff analysis and log analysis for defect localization, and exploring knowledge‑graph‑driven model‑based testing.