Building a Content QA Agent: Architecture, Prompts, and Evaluation Loop

Most content pipelines have one QA step: a human reads the draft before it goes live. That human is tired, context-switching, and working from a vague style guide written three years ago. They catch obvious problems and miss subtle ones — a stat that's outdated, a product claim that's been superseded, a tone that drifts from the brand voice mid-article.

A content QA agent doesn't replace editorial judgment. It handles the systematic checks that humans are bad at: cross-referencing facts against a source document, flagging every deviation from a defined style rubric, scoring brand alignment on a consistent scale. When it's built correctly, the human reviewer sees a report — not raw prose — and makes faster, better decisions.

This post walks through the architecture, the prompts, and the evaluation loop that makes a content QA agent production-ready.

What the Agent Actually Does

The agent takes a draft as input and returns a structured QA report. That report covers three categories:

1. Factual accuracy — Are claims supported by a provided source set? Does the draft introduce numbers or dates not present in the briefing materials?
2. Tone and style — Does the writing match the defined voice rubric? Are prohibited phrases present?
3. Brand alignment — Do product names, pricing, and feature descriptions match the authoritative company fact file?

Each category gets a pass/flag/fail verdict, a confidence score, and a list of specific line-level issues with evidence. The human editor reviews the report, not the raw draft. That single change cuts review time dramatically in our engagements — reviewers spend time on judgment calls, not manual cross-checks.

Architecture Overview

The agent runs as a multi-step pipeline, not a single prompt. Each step is a discrete LLM call (or a retrieval operation) with its own context window and evaluation criteria. Trying to stuff all three QA categories into one prompt produces worse results: the model trades off between tasks and produces shallow coverage on all of them.

Draft Input
    │
    ▼
[Step 1] Document Chunking & Retrieval Setup
    │
    ▼
[Step 2] Factual Accuracy Node  ←── Source Documents (RAG)
    │
    ▼
[Step 3] Tone & Style Node  ←── Style Rubric Document
    │
    ▼
[Step 4] Brand Alignment Node  ←── Company Fact File (e.g., COMPANY.md)
    │
    ▼
[Step 5] Report Synthesis Node
    │
    ▼
Structured QA Report (JSON)
    │
    ▼
Human Editor Review Interface

Each node writes its output to a shared state object. The synthesis node reads all three outputs and produces the final report. This is a linear chain — not a loop — on the first pass. The evaluation loop runs separately, which I'll cover below.

The orchestration layer can be LangGraph, a custom Python state machine, or even a simple sequential script if your volume is low. The specific framework matters less than the principle: one responsibility per node.

The Three Core Prompts

Factual Accuracy Prompt

This node uses retrieval-augmented generation. Before the LLM call, you retrieve the top-k chunks from your source document set (research PDFs, product docs, briefing materials) using semantic similarity against the draft's claims.

System:
You are a fact-checking assistant. You are given a content draft and a set of retrieved source passages. Your job is to identify every factual claim in the draft — statistics, dates, product specifications, named entities — and classify each as:
  - SUPPORTED: the claim appears in or is directly entailed by the source passages
  - UNSUPPORTED: the claim is present but cannot be verified against the sources
  - CONTRADICTED: the claim conflicts with information in the sources

For each finding, quote the exact draft sentence, state your classification, and if applicable quote the relevant source passage.

Return your response as a JSON array of findings.

User:
DRAFT:
{draft_text}

SOURCE PASSAGES:
{retrieved_chunks}

The key instruction is quote the exact sentence. Vague references ("the third paragraph") are useless to editors. Line-level evidence is what makes the report actionable.

Tone and Style Prompt

This node does not need RAG. It needs a well-structured rubric document passed directly in context.

System:
You are a brand voice reviewer. You are given a content draft and a style rubric. Your job is to identify passages that deviate from the defined voice. Score the draft on each rubric dimension from 1–5 (5 = fully aligned). For every score below 4, provide a specific quote from the draft and a brief explanation of the deviation.

Do not rewrite the draft. Only identify and explain problems.

Return a JSON object with:
  - dimension_scores: object mapping each rubric dimension to its score
  - overall_tone_score: weighted average
  - flagged_passages: array of {quote, dimension, issue_description}

User:
DRAFT:
{draft_text}

STYLE RUBRIC:
{rubric_text}

Your style rubric needs to be explicit, not aspirational. "Confident and direct" is not enough. The rubric should list prohibited phrases, define what "direct" means with examples, and specify where hedging is acceptable versus where it's not. Vague rubrics produce vague feedback.

Brand Alignment Prompt

This node compares the draft against your authoritative fact file — the single source of truth for product names, pricing tiers, feature descriptions, and team details.

System:
You are a brand compliance reviewer. You are given a content draft and a company fact file. Your job is to identify any place where the draft references company-specific information — product names, pricing, team members, service descriptions, URLs, case studies — and verify that each reference matches the fact file exactly.

Flag any discrepancy, even minor ones (e.g., "15 apps" vs "15+ apps", outdated pricing ranges, deprecated service names).

Return a JSON array of findings: {draft_quote, fact_file_reference, discrepancy_description, severity: "minor"|"major"|"critical"}.

User:
DRAFT:
{draft_text}

COMPANY FACT FILE:
{fact_file_text}

Severity classification matters. A deprecated service page URL is critical — it affects live links. A stylistic inconsistency in how a team member's name is formatted is minor. Editors need to triage quickly.

The Evaluation Loop

A single-pass agent catches most issues. An evaluation loop catches the agent's own mistakes.

After the three nodes complete, a fourth "meta-evaluation" node reviews the QA report itself for three failure modes:

1. Hallucinated citations — Did the factual accuracy node cite a source passage that doesn't actually support its verdict?
2. False positives — Did the tone node flag passages that are actually compliant with the rubric?
3. Missed critical issues — Does the draft contain obvious problems (factual contradictions, broken product claims) that none of the three nodes flagged?

This is where the evaluation loop earns its cost. Without it, editors start to distrust the agent because they occasionally catch errors the agent missed or get flooded with false positives. Either outcome kills adoption.

The meta-evaluation prompt is simpler than it sounds:

System:
You are reviewing a QA report generated by an automated content review agent. You are given the original draft, the source documents, the style rubric, the company fact file, and the agent's QA report. Your job is to:

1. Spot-check 3–5 factual findings and verify whether the cited source passage actually supports the verdict.
2. Review flagged tone passages and confirm whether the rubric citation is accurate.
3. Skim the draft for any obvious issues the report does not mention.

Return a JSON object with:
  - verified_findings: array of finding IDs confirmed correct
  - disputed_findings: array of {finding_id, reason_for_dispute}
  - missed_issues: array of {draft_quote, issue_description}

The meta-evaluation node runs on a sampled basis — not every draft, every time. In high-volume pipelines, running it on 20–30% of drafts gives you enough signal to catch systematic agent errors without doubling your token cost.

For a deeper look at how agents fail in production and how to build detection into your pipeline, see Agent Failure Modes: What Breaks Custom AI Agents in Production.

Choosing Your Infrastructure

Component	Lightweight Option	Production Option
Orchestration	Sequential Python script	LangGraph state machine
LLM	GPT-4o (single provider)	GPT-4o + fallback to Claude 3.5
Retrieval	In-memory FAISS	Pinecone or pgvector
State storage	Local dict / JSON file	Redis or Postgres
Report delivery	Slack message	Custom editor UI or CMS webhook
Evaluation loop	Manual spot-checks	Automated on 20–30% sample

For most teams shipping fewer than 50 pieces of content per month, the lightweight column is sufficient. The production column makes sense once you're running the agent on multiple content types (blog posts, ad copy, product descriptions) or integrating it into a CMS publishing workflow.

Building AI-powered content or product workflows? Semnexus's app development team has shipped custom AI agent architectures across multiple industries. If you're moving from prototype to production, we can help you get there without the usual false starts.

Observability: Knowing When It Breaks

Every node should log its inputs, outputs, and token usage. When an editor disputes a finding, you need to replay the exact call that produced it — not guess based on the current prompt version.

Minimum logging per node:

• Node name and run ID
• Input token count
• Output (full JSON, not truncated)
• Model version and temperature setting
• Latency

Track false positive rate and missed issue rate over time. If either metric degrades, it's usually a prompt drift problem — someone edited the rubric or fact file, and the prompt wasn't updated to match. This is more common than model regressions.

For a broader treatment of agent observability, AI Agent Observability: How to Know Your Agent Is Broken covers the instrumentation patterns in detail.

FAQ

What LLM should I use for a content QA agent?

GPT-4o is the default choice for instruction-following tasks with structured JSON output. Claude 3.5 Sonnet is a reasonable alternative and often produces cleaner JSON. Avoid smaller models for the factual accuracy node — precision on citation verification degrades noticeably below GPT-4-class capability.

How long does a single QA run take?

Typically 30–90 seconds for a 1,500-word draft, depending on retrieval latency and whether you're running nodes in parallel. For most editorial workflows, that's fast enough to run before human review. If you need sub-10-second turnaround, parallelize the three core nodes.

Can this agent replace a human editor?

No, and it shouldn't try to. The agent handles systematic, rule-based checks. Judgment calls — whether a paragraph is persuasive, whether an example resonates with the audience — stay with humans. The goal is to make human review faster and more focused, not to eliminate it.

How do I handle hallucinations in the factual accuracy node?

The meta-evaluation loop catches most of them. Additionally, lower the temperature on the factual accuracy node (0.0–0.2) and explicitly instruct the model to only classify a claim as SUPPORTED if it can quote the supporting passage verbatim. Forcing citation evidence significantly reduces confabulation.

What goes in the style rubric?

At minimum: prohibited phrases, required tone descriptors with examples, rules on contractions and first-person voice, guidance on sentence length, and any domain-specific language conventions. A rubric under 500 words is usually too vague. A rubric over 2,000 words starts to overwhelm the context window and confuse the model. Aim for 800–1,200 words of specific, example-driven guidance.

How do I version-control the prompts and rubrics?

Treat them as code. Store prompts in your repo alongside the agent code. Tag rubric versions and fact file versions so you can replay any historical run against the exact inputs that produced it. When a prompt changes, document what changed and why — prompt archaeology is genuinely hard without that trail.

---

If you're ready to move beyond prototype and ship a content QA agent — or any other custom AI workflow — into production, book a 30-minute call or explore what Semnexus's app development team has built for teams in similar situations. We'll tell you honestly whether this is a two-week build or a two-month one.