What Emerging Architectures Power Modern LLM Applications?

Based on interviews with AI startup founders and engineers, this a16z analysis presents a reference architecture for large‑language‑model (LLM) applications, showing the most common systems, tools, and design patterns observed in both startups and leading tech companies.

Design pattern: Context learning – the core idea is to use a pre‑trained LLM without fine‑tuning and steer its behavior by cleverly prompting it with a small set of the most relevant private “context” documents. The workflow is divided into three stages:

Data preprocessing / embedding : private data (e.g., legal documents) are chunked, passed through an embedding model, and stored in a vector database for later retrieval.

Prompt construction / retrieval : when a user query arrives, the application builds a prompt that combines hard‑coded templates, few‑shot examples, any external API data, and the set of relevant documents retrieved from the vector store.

Prompt execution / inference : the compiled prompt is sent to a pre‑trained LLM (proprietary API or self‑hosted model). Developers often add logging, caching, and validation at this stage.

The authors argue that this pattern is easier to adopt than training or fine‑tuning an LLM because it reduces the need for a dedicated ML engineering team and expensive dedicated infrastructure.

Vector databases are highlighted as the most critical piece of the preprocessing pipeline. Pinecone is the default choice because it is fully managed and offers enterprise‑grade features. Open‑source alternatives such as Weaviate, Vespa, and Qdrant provide strong single‑node performance and customizability, while local libraries like Chroma and Faiss are useful for small experiments. PostgreSQL extensions (e.g., pgvector) are mentioned as a viable option for teams that prefer an OLTP‑centric solution.

Embedding models – most developers use OpenAI’s ada‑002 text‑embedding model for its ease of use and cost‑effectiveness. Some large enterprises experiment with Cohere for potentially better performance in specific scenarios, and open‑source practitioners rely on Hugging Face’s sentence‑transformers library.

Orchestration frameworks such as LangChain and LlamaIndex abstract away prompt‑chain details, API calls, and vector‑store retrieval. LangChain (currently at version 0.0.201) is identified as the market leader, though some early adopters prefer raw Python for production stability.

Model choices and scaling trade‑offs – the majority of interviewees start with OpenAI’s GPT‑4 or GPT‑4‑32k for best performance. Cost considerations lead many to switch to gpt‑3.5‑turbo (≈50× cheaper) when lower latency and price are more important than top‑tier accuracy. Experiments with Anthropic’s Claude are noted for its fast inference and larger context windows (up to 100 k tokens). Open‑source models (Meta’s LLaMA, LLaMA 2, Falcon, Mistral, etc.) are gaining ground, and hosted services like Replicate are making them easier to consume.

Operational tooling – caching (often Redis) is common to reduce latency and cost. Monitoring and evaluation tools such as Weights & Biases, MLflow, PromptLayer, and Helicone are widely used to track LLM outputs, support prompt‑engineering iterations, and implement guardrails against hallucinations or prompt injection.

Hosting – static parts of LLM applications (frontend, APIs) are typically deployed on Vercel or major cloud providers. Emerging hosted‑LLM platforms (Steamship, Anyscale, Modal) offer end‑to‑end solutions that combine orchestration, multi‑tenant data contexts, and model hosting.

Agents – the reference stack omits AI‑agent frameworks, which the authors consider the next critical component. AutoGPT is cited as the fastest‑growing GitHub repo, described as an experimental attempt to make GPT‑4 fully autonomous. While agents promise capabilities such as complex problem solving, external tool use, and self‑improvement, most are still at proof‑of‑concept stage.

Future outlook – as LLM context windows expand, the role of embeddings and vector databases may evolve. Some anticipate embeddings becoming less important, but expert feedback suggests they will remain vital because larger windows increase compute cost, making efficient retrieval essential. Prompt‑engineering techniques (chain‑of‑thought, self‑consistency, generated knowledge, etc.) are expected to grow in sophistication, and orchestration frameworks will continue to abstract these patterns for developers.

Overall, the stack presented serves as a starting point for LLM application development, with the expectation that components will shift as foundational models, tooling, and hosting options mature.