Every Monday, an agency analyst exports last week's campaign results, opens a handful of client and competitor websites, and copies notes from a shared spreadsheet into a report. The account manager checks the numbers, rewrites the recommendations and sends the document back for corrections.
Your engineering team is asked to automate it. A prototype can collect some text and generate a convincing report. The harder questions appear when an API times out, someone edits the spreadsheet during a run, or a recommendation cites a website that has since changed. Which inputs produced the report? Can the reviewer check them? What happens when the job is retried?
I would treat this as a data and document pipeline with explicit stages. AI can help interpret evidence and write a narrative, but the application must control collection, calculations, validation and release.
The agency workflow below is an illustrative design. My AI Pitch Deck Generator demonstrates research enrichment, evaluation, revision and editable document generation. Lurkee demonstrates website collection, change detection and structured analysis. Neither case is presented here as an existing integration of all three input types or a measured client outcome.
Define the report before choosing the agent
Start with one report the team already uses. Identify its readers, the decisions it supports, its reporting period and the person responsible for approval. Separate required sections from optional research.
For this example, the report contains campaign performance, relevant market observations and recommendations. The campaign totals come from an API export. Approved campaign IDs and client context come from a spreadsheet. Selected website passages support the market observations. These sources have different responsibilities; they should not silently replace one another.
Agree on the comparison rules before implementation. Does “last week” mean the client's timezone or the API account's timezone? Are we comparing completed weeks? Which currency and conversion definition apply? A report can be fluent and still compare incompatible periods.
I would begin with a fixed pipeline and a small source allowlist. An agent that discovers additional pages may be useful for exploratory research, but it introduces variability in coverage and cost. Keep that exploration in a separate section, with its own limits and review requirements, until the team can evaluate it reliably.
Make each run a record of its inputs
Create a run record before collection. It holds the client scope, reporting period, source list and versions of the extraction rules and report template. Connectors execute within that scope, using permissions checked by the application.
For an API, preserve the requested filters, pagination coverage and returned data. For a website, retain the fetched content or permitted evidence excerpt, retrieval time and extraction status. For a spreadsheet, capture the selected values and range rather than letting later stages reread a changing sheet. Retention and storage access should match the client's data policy.
Saving inputs prevents later edits from silently changing the report's evidence. It does not make independent sources a simultaneous snapshot. Record when each source was read and reject or flag combinations outside the agreed freshness window.
Give each usable input a source ID. A passage should have a locator, such as a stored section or row range, so the reviewer can find the evidence without repeating the research. Also record collection failures. A missing API page is a coverage problem, not proof that no campaigns ran.
If an optional market page is unavailable, the draft may proceed with a named limitation. If the campaign API is incomplete, I would block the performance section. Those are report-specific policies, not decisions left to the model.
Calculate first, interpret second
Compute totals, ratios and period comparisons in ordinary application code. Define behaviour for missing values, duplicate rows, a zero denominator and currency mismatches. The model receives those calculated results with their units and provenance; it does not improvise arithmetic from a mixture of pages and spreadsheet text.
Consider a hypothetical result: spend rose while the recorded number of conversions fell. The pipeline can calculate that comparison from compatible periods. It cannot establish that a competitor's new landing page caused the change merely because both appear in the evidence bundle.
Ask the analysis stage for structured findings: an observation, supporting source IDs, relevant values, limitations and a separately labelled interpretation. Validate the output's shape, then check the evidence. A valid JSON object does not establish that a claim is true.
Useful checks include resolving every source ID, confirming quoted passages exist in the stored input and matching numerical claims to the calculation results. Those checks catch identifiable mismatches; they do not settle every question of meaning or causation. Unsupported explanations stay out of the final narrative or go to review as hypotheses.
Retrieved pages and spreadsheet cells are untrusted input. A sentence inside them must not grant permission to change the report recipient, fetch another client's records or send a document. Keep credentials and delivery controls outside the model's context and enforce source and client boundaries in code. Delimiters and prompt instructions can help, but they do not replace those controls.
Give the reviewer something better than a PDF to proofread
The review screen should show the proposed finding next to its evidence and calculation. The reviewer can accept it, edit the interpretation, request another source or exclude it. Track which version they reviewed.
Generate narrative sections from accepted findings, then check that drafting has not introduced new numbers or unsupported claims. A model can add an explanation during rewriting even when the initial findings were well constrained. Changes to a calculation or evidence bundle should invalidate the affected approval.
Render the reviewed content into the agreed template. Keep headings, tables and citations as document elements rather than one flattened image. If the team edits PowerPoint or Word, produce that editable format; a PDF can be a separate presentation copy.
This introduces an ownership question: if someone edits the exported document, is it now the final version? For an initial release, I would designate the review application as the approval record and label exports with their report version. Importing arbitrary offline edits back into the pipeline is a separate feature, not an assumed capability.
Retry a stage without restarting the entire process
Collection, analysis and rendering fail differently. A temporary API failure may warrant a bounded retry. Invalid credentials need operator attention. An unsupported model finding needs revision or review. A rendering failure should not require the analyst to pay for all the research again.
Persist stage outputs and attempt records. Identify work by run, stage, input digest and configuration version. For a repeated attempt with unchanged inputs, reuse a completed artifact where the stage permits it. Concurrent workers need an atomic claim or uniqueness rule, not just a check followed by an insert.
If a worker crashes after saving an artifact but before reporting success, the retry should locate that artifact and complete the stage record. If the source values or template change, create a new version and rerun the dependent stages. The goal is controlled recovery, not a claim that every task executes exactly once.
Delivery needs its own record. If sending times out after a provider may have accepted the document, reconcile the provider's status where supported before retrying. If acceptance cannot be established, expose the uncertain state to the operator. A successfully rendered file is not evidence of successful delivery.
Test the report against difficult inputs
Use representative historical inputs with the team's agreed calculations and expected evidence. Evaluate usefulness with the people who review the report, rather than relying only on another model's score.
| Test condition | Expected behaviour |
|---|---|
| API returns only part of the required period | Coverage check blocks the affected section |
| A sheet changes after collection | The current run uses its captured values; a refresh creates a new version |
| A calculation has a zero denominator | The report shows an agreed unavailable value, not an invented percentage |
| A finding cites an absent passage | Validation rejects the reference and requests revision or review |
| A website contains instructions to send data elsewhere | Application permissions prevent that source text from controlling delivery |
| Rendering fails after analysis completes | Recovery reuses the recorded analysis for unchanged inputs |
| Evidence changes after approval | The affected report version requires review again |
| Delivery status is uncertain | The operator sees uncertainty; the system avoids a blind resend |
Track collection completeness, unsupported findings, reviewer corrections, run duration and cost per approved report. Count retries and human review time as part of the workflow. Set targets from the existing process and the team's needs; generating more words is not a useful success metric.
What I would deliver to the engineering team
A first release should cover one recurring report, an agreed set of sources and one editable output format. I would hand over the source contracts, stage interfaces, evidence model, review rules, recovery procedures and representative acceptance cases alongside the application.
The AI Pitch Deck Generator shows my approach to separating research, narrative, evaluation and rendering. Lurkee shows the collection and structured-analysis side. For the related problem of deciding which website changes deserve attention, see competitor monitoring without noisy alerts.
If your engineering team is building research automation or document generation and needs someone to own the path from source integration through review and delivery, tell me about the workflow and the role you need to fill. Bring an example report, its input sources and a failure the current process handles poorly. For a scoped implementation, my AI product engineering service covers the same work from integration to an operable product.