Your team follows competitor pricing, product pages and announcements. At first, a few page-change emails are useful. As the watch list grows, rotating banners, reordered products and revised footers start filling the inbox. Someone still has to open each page to decide whether anything matters.
Adding AI to every check can make that process more expensive without making it more useful. The engineering problem is to collect reliable evidence, isolate relevant changes and deliver findings that support a decision.
Consider a product team that wants to know when a competitor changes a plan price, moves a feature between tiers or introduces a new offer. I would build the monitoring around those questions. The workflow below is an illustrative project design informed by my work on Lurkee, rather than a report of a client deployment or measured savings.
Define what deserves the team's attention
“Monitor these websites” leaves too much undecided. For a pricing page, we need to agree on the region, currency, billing period and plan we are comparing. For an announcement page, we need to distinguish a new release from an edit to an old post.
I would start with a small source list and examples of useful and unwanted notifications. A pricing change might need prompt review by the product owner. A revised headline could belong in a weekly digest. A timestamp or cookie banner should usually be excluded from the business comparison.
Those examples become monitoring rules: which fields to collect, how often to check them, what counts as a meaningful change and who receives it. They also define coverage. Watching a pricing page does not automatically cover its regional versions, checkout discounts or the rest of the product catalogue.
Establish a change before asking AI to explain it
The collection stage retrieves a page and checks whether it contains the expected content. It then extracts the fields or sections we intend to monitor. For a pricing page, that could be plan names, prices, billing labels and included features.
I would normalize predictable noise before comparing snapshots: whitespace, irrelevant navigation and known rotating elements. The rules must preserve business information. Removing every number would hide price changes; sorting every list could hide a meaningful ranking change.
A content hash provides a quick equality check for that normalized representation. If the representation is unchanged, the check can finish without an AI call. If it differs, a diff identifies the changed fields or passages. The model receives those changes with enough surrounding context to interpret them.
The first valid snapshot establishes a baseline. It should not be described as a newly discovered change. When an extraction rule changes, I would version it and handle the new baseline explicitly, so a parser update does not masquerade as a competitor announcement.
This separation reduces unnecessary model calls. It does not make collection free: fetching, browser rendering, storage and retries still contribute to operating costs.
Turn the diff into a finding someone can verify
Suppose a watched plan changes from $49 to $59 per month. That is a hypothetical example. Before raising a price-change alert, the system should confirm that both snapshots refer to the same plan, currency and billing period. Comparing an annual equivalent with a monthly price would produce a misleading finding.
I would ask the analysis stage for a structured result: change category, observed before and after values, source references, reason for relevance and any uncertainty. The backend checks required fields and verifies that quoted evidence exists in the stored snapshots. A schema-valid answer can still contain an unsupported interpretation.
The finding should distinguish observation from inference. “The monthly price increased” can be supported by the comparison. “The competitor is moving upmarket” is an interpretation that needs more evidence.
Importance also depends on the client. The same change may deserve an immediate alert for a pricing team and a digest entry for an agency tracking broader market activity. I would apply explicit routing rules after validating the finding. An unresolved billing-period mismatch should go to review, rather than receiving a confident price-change label.
Repeated delivery needs its own control. I would identify a finding by its source and snapshot pair, record delivery attempts and use duplicate protection when retrying notifications. A repeated check of the same change should not create a fresh business event every time.
Keep collection failures separate from business changes
A timeout, blocked request or missing pricing section tells us that the check did not succeed. Comparing an empty extraction with yesterday's page could incorrectly report that every plan disappeared.
I would retain the last successful baseline and record the failed attempt separately. The dashboard should show both when collection was last attempted and when usable evidence was last captured. “No change detected” is appropriate only after a successful comparison; a failed check means current coverage is unknown.
Retries should have limits and increasing delays. I would constrain concurrent requests per source and record recurring extraction failures for operator review. A growing queue or stale source is an operational issue that needs visibility, even when no business alert is being sent.
Some sources require browser rendering; others can be collected with simpler requests or an available feed. That choice should follow the source's behaviour and permitted access. Unsupported or inaccessible sources should appear as coverage gaps in the agreed scope.
Control costs at each stage of the pipeline
The operating budget should include collection, rendering, stored evidence, AI analysis and delivery. A useful estimate separates scheduled checks, successful comparisons, changed snapshots and changes requiring deeper analysis.
For model usage, a simple planning relationship is:
AI cost ≈ changed snapshots × average analysis cost + escalated findings × additional review cost.
This is a budgeting model, not a forecast. Retries and reprocessing add work; actual costs need to be measured against representative sources and provider usage.
I would limit input to the relevant diff and context, reuse results for the same comparison and analysis version, and reserve more expensive reasoning for cases that need it. A lower-cost first pass can miss something important. Model routing therefore needs evaluation on examples the client considers critical, alongside cases likely to produce noise.
Check frequency is another lever. A release archive and a volatile pricing page may need different schedules. Shorter intervals can improve detection time while increasing requests and processing load. The right balance depends on how quickly the team can act on the result.
Prove usefulness before expanding coverage
I would begin with a representative set of sources and a review period in which the team labels findings as useful, irrelevant or incorrect. Known changes and collection failures should be included in the evaluation, rather than judging the system only by how many alerts it sends.
| Scenario | Expected behaviour |
|---|---|
| Valid normalized content is unchanged | Record a successful check; skip AI analysis |
| Same-plan price changes | Preserve both values and source evidence; apply the client's alert rule |
| Only an excluded banner changes | Keep it out of the business comparison |
| Billing period cannot be matched | Mark uncertainty and send for review |
| Required page section is missing | Record collection failure; retain the last valid baseline |
| Notification delivery is retried | Track the attempt and avoid creating a duplicate finding |
During the pilot, I would measure successful coverage, evidence age, incorrect alerts, known changes missed, queue delays and cost per useful finding. These measurements tell us whether to revise extraction rules, model routing or scheduling before adding more sources.
A custom system makes sense when the team needs domain-specific comparisons, shared history, review workflows or integration with its own product. A simpler page-change tool may be enough for a small watch list. That decision should follow the workflow requirements and expected operating burden.
What Lurkee demonstrates
In Lurkee, I built a FastAPI monitoring pipeline with scheduled collection, content hashing and change detection before AI interpretation. Structured severity and findings feed reports, alerts and an operator dashboard. Routine analysis can use a lower-cost model, with stronger reasoning reserved for findings that cross a severity threshold.
Those are portfolio implementation details. The source-specific rules, failure policies and acceptance scenarios described here are how I would scope a new monitoring project.
Build around the decision the monitoring should support
A useful competitor monitoring system lets someone answer: what changed, what evidence supports it, why might it matter and what should we do next? It also makes clear when a source could not be checked.
I offer AI product engineering for custom monitoring pipelines, operator dashboards and integrations with existing products.
If you want to build or improve one, send me the sources you need to monitor, a useful alert example and the decision your team wants to make. If a system already exists, include a noisy alert and its current collection process. We can use that to define a first implementation with clear coverage, review criteria and an operating budget.