How to Build a Competitor Monitoring Agent With Weekly Digest Output
Competitive intelligence work is repetitive, time-consuming, and easy to deprioritize. Most teams agree they should be watching competitor pricing pages, blog posts, app store listings, and social channels — and most teams do it sporadically, if at all. A custom AI agent built specifically for competitor monitoring solves this by converting a weekly manual chore into a scheduled, automated digest that lands in your inbox (or Slack) every Monday morning.
This guide walks through the architecture, tooling choices, prompt design, and failure modes of a working competitor monitoring agent. You'll be able to implement this yourself, or hand it to your engineering team as a spec.
What the Agent Actually Does
Before touching architecture, be precise about the job. This agent does five things:
- Crawls a defined set of competitor URLs on a schedule (weekly by default)
- Diffs the current content against a stored snapshot to detect changes
- Classifies each change by type — pricing, feature announcement, messaging, job posting, app store listing
- Summarizes changes using an LLM prompt tuned for competitive relevance
- Composes and delivers a structured weekly digest to a configured destination
That's it. The agent doesn't make decisions or file tickets. It observes, classifies, and reports. Keeping the scope tight is what makes it reliable. Scope creep — adding sentiment scoring, automated response drafting, Salesforce sync — is where these agents start to break. If you're curious about how scope growth introduces failure modes, read our breakdown of what breaks custom AI agents in production.
Choosing Your Stack
You have three realistic options for the core infrastructure:
| Layer | Lightweight Option | Mid-tier Option | Full-stack Option |
|---|---|---|---|
| Scheduling | GitHub Actions (cron) | AWS EventBridge | Temporal.io |
| Scraping | Playwright + Cheerio | Browserless.io | Apify |
| Storage / Snapshots | S3 + JSON | PostgreSQL | Supabase |
| LLM | GPT-4o-mini | GPT-4o | Claude Sonnet |
| Digest delivery | Resend (email) | Slack Webhook | Both |
| Orchestration | Plain Node.js | LangChain | LangGraph |
For most teams running this as an internal tool, the lightweight column is enough. A GitHub Actions cron job, Playwright for rendering JavaScript-heavy pages, S3 for snapshot storage, GPT-4o-mini for summarization, and Resend for delivery. The total infrastructure cost is typically under $30/month for a watchlist of 10–20 competitor URLs. For a more detailed breakdown of per-run costs, see our AI agent cost modeling post.
If your competitors have heavy anti-bot protection (Cloudflare, reCAPTCHA, heavy JS SPAs), budget for Browserless.io or Apify instead of raw Playwright. The scraping layer is where most teams hit unexpected friction.
Building the Crawl and Diff Layer
The crawl layer has two jobs: fetch the current page content and produce a clean, diffable representation of it.
Don't diff raw HTML. Raw HTML diffs are noisy — session tokens, ad slots, timestamps, and CDN-injected attributes will fire false positives on every run. Instead, strip the HTML down to meaningful text content before storing the snapshot.
Here's the pattern in Node.js:
import { chromium } from 'playwright';
import { convert } from 'html-to-text';
import crypto from 'crypto';
async function fetchPageContent(url) {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle' });
const html = await page.content();
await browser.close();
const text = convert(html, {
wordwrap: false,
selectors: [
{ selector: 'nav', format: 'skip' },
{ selector: 'footer', format: 'skip' },
{ selector: 'script', format: 'skip' },
{ selector: 'style', format: 'skip' },
],
});
const hash = crypto.createHash('sha256').update(text).digest('hex');
return { text, hash };
}
Store the hash and the cleaned text in S3 as snapshots/{competitor_slug}/{url_slug}/latest.json. On each run, fetch the new content, compare hashes, and only pass changed pages to the LLM. This keeps your token spend proportional to actual changes, not total crawl volume.
One important edge case: some competitors version their pages through redirects or canonical URL changes. Track the final resolved URL, not the input URL, to avoid treating a redirect as a content change.
Designing the Classification and Summary Prompt
This is where most implementations get lazy and produce useless digests. A generic prompt like "Summarize what changed on this page" generates summaries that don't tell you anything actionable. The prompt needs to encode what your team actually cares about.
A working classification prompt structure:
You are a competitive intelligence analyst. You will be given the previous and current text of a competitor's webpage.
Your job:
1. Identify the type of change from this list: PRICING, FEATURE_ANNOUNCEMENT, MESSAGING_SHIFT, NEW_PRODUCT, HIRING_SIGNAL, APP_STORE_UPDATE, OTHER
2. Write a 2-3 sentence summary of what changed, written for a product or marketing leader at a competing company.
3. Assign a relevance score from 1–5, where 5 = immediate strategic implication, 1 = cosmetic/minor.
4. Note any specific numbers, claims, or named features that changed.
Return JSON with keys: change_type, summary, relevance_score, notable_details.
Do not speculate. Only report what changed. If there is no meaningful change, return relevance_score: 0.
The relevance_score: 0 escape hatch is important. Without it, the LLM will manufacture summaries for pages that haven't changed meaningfully — a classic hallucination trigger when you're forcing output on every diff.
Filter out any result with relevance_score of 0 or 1 before building the digest. Your team will stop reading the digest the moment it fills with noise.
Composing the Weekly Digest
The digest composer is a second LLM call that takes the week's classified changes and writes a coherent briefing. This separation matters — classification and synthesis are different tasks, and combining them into one prompt produces worse output on both dimensions.
The digest prompt should produce a structured output your team can skim in two minutes:
- Top signal this week (one sentence, highest relevance-score item)
- Change summary table (competitor, URL, change type, relevance score, 1-line summary)
- Full summaries for items with relevance score 4–5 only
- Low-signal items listed without elaboration
Deliver it as an HTML email via Resend (or a Slack Block Kit message if your team lives in Slack). The HTML email format has one advantage: you can link directly to the competitor URL in the summary table, so readers can verify anything that catches their eye.
Scheduling, Storage, and Retry Logic
The cron schedule is straightforward — a GitHub Actions workflow on schedule: cron: '0 7 * * 1' fires every Monday at 7am UTC, which puts it in inboxes before the US workday starts.
What most teams skip: retry logic and idempotency. If a scrape fails mid-run (a competitor's site is down, Playwright times out), you want the agent to log the failure, skip that URL for this run, and include a note in the digest rather than silently dropping the competitor from coverage.
Store a run log in S3 — runs/{YYYY-MM-DD}/run_log.json — with per-URL status: success, scrape_error, no_change, classified. The digest should include a one-line footer: "3 URLs had scrape errors this week — check run log for details." Transparency about coverage gaps is more useful than a clean-looking digest that's missing data.
Snapshot retention: keep 4 weeks of snapshots per URL. This lets you answer "when did they change their pricing page?" without querying an external source.
Extending to App Store Listings
If you're competing in mobile, your competitor watchlist should include App Store and Google Play listings — not just web pages. App store listings change frequently: screenshots, descriptions, keywords (inferred), ratings, and update notes all carry competitive signal.
The App Store and Google Play both have unofficial APIs and scraping endpoints. For App Store, itunes.apple.com/lookup?id={app_id}&country=us returns structured JSON including the current description, version, and rating. No Playwright required. Store the parsed fields directly rather than diffing HTML — cleaner diffs, more reliable classification.
For example, a competitor dropping from a 4.6 to a 3.9 rating over a three-week span is a signal worth flagging. A new "What's New" note mentioning a specific integration is a feature announcement. These are easy to classify with the same prompt structure above.
If you're running mobile app marketing and want competitive intelligence built into your growth workflow, our mobile app marketing services team can help you operationalize what you're tracking into actual strategy.
FAQ
How many competitor URLs can this agent handle per run?
Practically, 20–50 URLs per weekly run is manageable with a GitHub Actions runner and a standard Playwright setup. Above 50, you'll want to parallelize crawls and consider a dedicated scraping service. The LLM cost scales with the number of changed pages, not total URLs crawled — so a larger watchlist doesn't automatically mean higher LLM spend.
What LLM should I use for this?
GPT-4o-mini handles classification and summarization well at a fraction of the cost of GPT-4o. For most competitive intelligence use cases, the quality difference doesn't justify the cost difference. Use GPT-4o only for the final digest composition step if you want more polished prose.
How do I handle competitor pages that require login?
You don't, for this pattern. Authenticated scraping introduces legal and ethical complexity that isn't worth it for a monitoring agent. Focus on public-facing pages: pricing, features, blog, app store listing, job postings. You'll get 80% of the useful signal from those.
Can this agent monitor social media and news mentions too?
Yes, but treat those as separate data sources with separate pipelines. Twitter/X search, Google News RSS, and Reddit can be polled with much simpler HTTP requests — no Playwright needed. Feed those results into the same classification and digest pipeline with a source: social or source: news tag so your team can filter by source type.
How do I prevent the agent from flagging the same change twice?
Once a change has been classified and included in a digest, mark the snapshot as the new baseline. The agent should only diff against the most recent post-delivery snapshot, not the original. This prevents a sticky change (like a revised pricing page) from re-appearing in every weekly digest until the competitor changes it again.
Is this hard to maintain over time?
The main maintenance burden is the competitor watchlist — URLs change, companies get acquired, pages move. Budget 30 minutes per month to audit the watchlist and update dead links. The agent itself, if built with the structure above, is largely self-maintaining. The run log makes problems visible without requiring you to babysit the process.
If you want a custom AI agent built and shipped rather than built yourself, our team has done this across competitive intelligence, sales automation, and support workflows. Start with a 30-minute call — book time here — or explore what our app development team can build for your specific use case.
