Scoping a RAG-Based Knowledge Agent: Architecture Decisions and Cost Ranges
Most internal knowledge problems aren't LLM problems. They're retrieval problems dressed up as LLM problems. A model that can reason brilliantly still fails when you feed it the wrong document chunks, or when your embedding pipeline treats a 40-page policy PDF the same way it treats a three-sentence Slack message. The architecture decisions you make in weeks one and two of an AI agent development project determine whether your knowledge agent is useful or quietly ignored by the team it was supposed to help.
This guide covers the structural decisions—retrieval design, chunking strategy, vector store selection, guardrails, and hosting—and gives you honest cost ranges at each layer. We'll focus on the internal knowledge use case: an agent that answers questions against your company's own documents, SOPs, product wikis, or support history.
What a RAG Agent Actually Does (and Where It Can Break)
Retrieval-augmented generation (RAG) chains two systems together: a retrieval system that finds relevant passages, and a language model that synthesizes an answer from those passages. Neither is optional. Strip out retrieval and you have a general-purpose chatbot hallucinating policy details. Strip out the LLM and you have a search bar.
The failure modes are worth naming upfront, because they drive the architecture choices later. The most common ones in our engagements:
- Retrieval misses — the right document exists but isn't returned because the query and the chunk don't share enough semantic overlap.
- Context window stuffing — too many chunks get passed to the model, diluting the signal and increasing cost per query.
- Stale embeddings — source documents update but the vector store doesn't re-index, so the agent confidently cites outdated policy.
- Chunk boundary failures — a chunk starts mid-sentence or cuts a table in half, and the model tries to reason over incomplete information.
If you want a deeper treatment of what breaks custom AI agents once they're live, the post on agent failure modes in production is worth reading before you finalize your architecture.
The Four Core Architecture Decisions
1. Chunking Strategy
Chunking is where most teams make their first mistake. Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is easy to implement and works tolerably for prose-heavy documents. It falls apart on structured content: tables, numbered lists, code blocks, and anything where the meaning of a sentence depends on surrounding context.
Better approaches for most internal knowledge bases:
- Semantic chunking — split on sentence boundaries, then merge small chunks until a coherent semantic unit is formed. Slower to build but meaningfully better retrieval.
- Document-aware chunking — parse markdown headers, PDF section boundaries, or Confluence page structure first, then chunk within sections. The section title stays attached to every chunk it produces.
- Hierarchical chunking — store both a summary chunk and its sub-chunks. Retrieve summaries first, then drill into sub-chunks only when needed. Higher storage cost, but dramatically better for long-form documents like 50-page compliance guides.
The right choice depends on your source material. A Notion wiki with consistent heading structure calls for document-aware chunking. An unstructured pile of uploaded PDFs calls for semantic or hierarchical.
2. Embedding Model Selection
Your embedding model converts text into vectors. The retrieval quality ceiling is set by the embedding model—a better reranker or larger LLM can't fix fundamentally poor embeddings.
| Model | Dimensions | Approximate Cost | Best For |
|---|---|---|---|
| OpenAI text-embedding-3-small | 1,536 | ~$0.02 / 1M tokens | General English content, fast iteration |
| OpenAI text-embedding-3-large | 3,072 | ~$0.13 / 1M tokens | Higher accuracy needs, multilingual |
| Cohere embed-v3 | 1,024 | ~$0.10 / 1M tokens | Enterprise retrieval, strong multilingual |
| open-source (e.g., BGE-M3, self-hosted) | varies | Compute cost only | Cost-sensitive, on-prem data requirements |
For most internal knowledge agents serving English-language content, text-embedding-3-small covers the baseline well. Move to large or Cohere when you're working with multilingual content or when retrieval accuracy testing shows meaningful gaps.
3. Vector Store Selection
The vector store holds your embeddings and handles the nearest-neighbor search. The right choice depends on scale, existing infrastructure, and whether you need metadata filtering.
- Pinecone — managed, fast, easy to start. Approximately $70–$150/month for a starter pod handling tens of millions of vectors. No infrastructure to manage.
- Weaviate — open-source or managed. Strong when you need hybrid search (vector + keyword BM25). Self-hosted option matters if data residency is a concern.
- pgvector — if you're already on PostgreSQL, adding the pgvector extension is the lowest-friction path. Retrieval speed degrades at very large scale but is fine for internal knowledge bases under ~5M vectors.
- Qdrant — high performance, strong filtering capabilities, self-hosted or cloud. Good middle ground between Pinecone's ease and pgvector's cost.
For most internal knowledge agents we scope, pgvector works until it doesn't—and it usually doesn't stop working until you're past a scale most internal tools never reach. Start there, migrate later if benchmarks show it.
4. Reranking
Raw vector search returns the top-k nearest neighbors by cosine similarity. Reranking runs a second, more expensive model over those candidates to reorder them by actual relevance. The retrieval improvement is typically significant enough to justify the latency and cost addition in any agent where answer quality matters.
Cohere Rerank and cross-encoder models (e.g., ms-marco-MiniLM variants, self-hosted) are the two common paths. Add reranking after you have a working baseline retrieval pipeline—not before. It's easier to benchmark the delta that way.
Orchestration and Agent Loop Design
RAG knowledge agents aren't just retrieval + LLM. Production agents need:
- Query rewriting — transform the user's raw question into a better retrieval query before hitting the vector store. A user asking "what's our refund window?" retrieves better when rewritten as "customer refund policy duration."
- Multi-step retrieval — for complex questions, the agent may need to retrieve, partially answer, identify gaps, and retrieve again before generating a final response.
- Citation grounding — every answer should reference the source chunk(s) it used. This isn't just good UX; it's the primary defense against hallucination in knowledge agents.
- Fallback handling — when retrieval returns nothing above a similarity threshold, the agent should say so explicitly rather than fabricate from parametric memory.
Frameworks like LangChain and LlamaIndex handle much of this scaffolding. LlamaIndex tends to be the stronger default for RAG-heavy workloads—its document ingestion and retrieval abstractions are more mature. LangChain gives you more flexibility for multi-agent orchestration when the knowledge agent is one node in a larger system.
Build Timeline and Cost Ranges
These ranges reflect the scope of a production-ready internal knowledge agent, not a demo.
| Phase | Duration | Approximate Cost Range |
|---|---|---|
| Discovery (document audit, chunking strategy, baseline retrieval testing) | 1–2 weeks | $4,000–$8,000 |
| Core build (ingestion pipeline, vector store, agent loop, API layer) | 3–5 weeks | $12,000–$35,000 |
| UI / chat interface (if needed) | 1–2 weeks | $4,000–$10,000 |
| Evaluation, reranking tuning, guardrails | 1–2 weeks | $5,000–$12,000 |
| Total build | 6–11 weeks | $25,000–$65,000 |
Ongoing infrastructure runs approximately $200–$800/month depending on vector store choice, LLM call volume, and whether you self-host any components. Per-query cost modeling matters here—we covered that in more detail in the AI agent cost modeling post.
Working on an internal knowledge agent and not sure where to start? The app development team at Semnexus has scoped and shipped custom AI agents across multiple industries—we can help you define the architecture before you commit to a build approach.
Guardrails and Data Hygiene
A knowledge agent is only as trustworthy as its source documents. Before you embed anything:
- Deduplicate — duplicate documents inflate retrieval noise. The same SOP updated three times means three versions fighting each other in the vector store.
- Version control your corpus — know which document version each chunk came from. When policy changes, you need to be able to re-index cleanly.
- Access-scope the retrieval — if some documents are HR-only and others are company-wide, your agent needs to filter by access level at query time, not at ingestion time. Filtering at ingestion (just don't embed the restricted docs) looks simpler but breaks down when the same user asks a question that spans access tiers.
- Confidence thresholding — set a minimum similarity score below which the agent refuses to answer from retrieval and says explicitly that it doesn't have a reliable source. Hallucination in a knowledge agent is worse than a "I don't know"—it erodes trust faster than anything else.
FAQ
How much data do I need before a RAG agent is worth building?
There's no hard floor, but approximately 50–100 substantial documents (SOPs, policies, product docs, support guides) is where retrieval starts outperforming manual search. Below that, a well-organized wiki with good search is often the right answer.
Can I use an off-the-shelf tool like Notion AI or Guru instead of building a custom agent?
Yes, and you should evaluate them first. Off-the-shelf tools are faster to deploy and cheaper to operate. Build custom when you need retrieval across heterogeneous sources (Confluence + Google Drive + a SQL database), when you need to integrate the agent into a larger workflow, or when access-scoping requirements are complex.
How do I evaluate retrieval quality before shipping?
Build an evaluation set: 50–100 representative questions with known correct answers and the source documents they should retrieve from. Measure hit rate (did the right document appear in the top-k results?) and answer correctness separately. The AI agent evaluation frameworks post covers how to structure this in detail.
What LLM should I use for the generation layer?
GPT-4o is the default choice for most internal knowledge agents—strong reasoning, reliable instruction-following, and well-understood behavior. Claude 3.5 Sonnet is a strong alternative, particularly when answer verbosity and citation formatting matter. Run both on your evaluation set before committing.
How do I handle documents that update frequently?
Build incremental re-indexing into your pipeline from day one. When a document changes, delete its existing chunks from the vector store by document ID and re-embed the updated version. This is significantly harder to retrofit than to design in upfront.
Is self-hosting the LLM ever worth it for this use case?
Rarely, for an internal knowledge agent. The compute cost and engineering overhead of running an open-source model (Llama 3, Mistral, etc.) at production quality typically exceeds the API cost savings unless you're processing millions of queries per month or have strict data residency requirements that rule out third-party APIs entirely.
If you're scoping a RAG-based knowledge agent and want a second opinion on the architecture before you start building, book a 30-minute call with Marco or review what the Semnexus app development team builds end-to-end. We'll tell you what we'd actually build—and where off-the-shelf tools might serve you better.
