Use an Evidence Gate Before Approving an AI Architecture

A team is being asked to make an AI architecture decision, but the supplied evidence does not document an AI model, a retrieval system, an agent, or even a defined user problem. One source is identified as a GOV.UK page associated in the metadata with physical activity guidance. The other is an AWS documentation page titled “Amazon Simple Storage Service,” supplied without a summary.

That is not enough to choose an AI architecture, estimate its economics, or approve implementation.

The immediate decision is therefore narrow: should an AI, RAG, or agent-based architecture be approved from the current evidence packet? No. The defensible action is to pause architecture approval, identify the missing claims, and request primary evidence for each one.

This is not a conclusion that AI is unsuitable. It is a conclusion that the recommendation is not yet supported.

What the supplied record actually verifies

The evidence packet contains two source records:

• The first record is titled “GOV.UK.” Its metadata names “UK Physical Activity Guidelines” and describes guidance from the UK Chief Medical Officers about physical activity. Its canonical URL is the GOV.UK homepage rather than a specific report page (GOV.UK).
• The second record points to AWS documentation and is titled “Amazon Simple Storage Service.” Its supplied summary is empty (AWS documentation).

Those are the verified facts available from the supplied metadata. They do not establish any model capability, model version, price, context limit, API behavior, retrieval quality, latency, security property, or expected business outcome.

There is also a source-quality issue inside the record itself. The GOV.UK source name and summary refer to physical activity guidance, while the canonical link and title identify only the GOV.UK root. That mismatch does not prove that the summary is wrong, but it prevents the canonical page from directly supporting the described claim.

The AWS record is relevant only at a broad infrastructure level. The metadata identifies Amazon Simple Storage Service documentation, but it supplies no specific S3 feature, behavior, limitation, API contract, or price. Treating that record as evidence for a detailed architecture would require facts that are not present here.

Separate the decision from the architecture vocabulary

Terms such as “RAG,” “agents,” and “automation” can make an incomplete proposal sound more concrete than it is. They do not answer the first product question: what user or business problem requires a system change?

A review should begin with a decision statement that does not prescribe the solution. For example:

Determine whether the proposed workflow requires probabilistic language processing, deterministic software, or a combination of both.

That statement is an illustrative review format, not a claim about a supplied product. The current metadata does not describe a workflow, users, inputs, or desired outcome.

Ordinary software should remain an explicit option. If the required behavior can be expressed as stable rules over structured data, a deterministic application may be easier to test and control. If the task depends on interpreting variable natural-language inputs, an AI component may become a candidate. These are decision criteria, not findings about this project.

Without a defined problem, selecting an AI pattern first would reverse the order of reasoning. It would also make evaluation arbitrary because there would be no agreed task or failure condition.

Build a claim-to-evidence matrix before design approval

A practical evidence gate maps every consequential claim to a source and a test. The current packet would look like this:

The value of the matrix is not bureaucracy. It prevents a source from being stretched beyond the claim it supports. General service documentation cannot establish model quality. A provider price page cannot establish task accuracy. An internal estimate cannot establish API behavior.

Each row also needs an owner and an invalidation condition. A capability claim should fail if the named model cannot meet the agreed evaluation threshold. A cost estimate should be recalculated if workload volume, prompt size, output size, retrieval frequency, human-review time, or provider pricing changes materially.

The technical path starts with evaluation, not integration

With the current evidence, implementation should not begin. A responsible technical path would require the following inputs before architecture selection.

1. Define the task boundary

Specify the exact input, expected output, permitted uncertainty, and consequence of an incorrect result. Separate tasks that can be handled by deterministic rules from tasks that may require model inference.

This boundary determines whether AI is necessary at all. It also limits the temptation to turn one uncertain component into an autonomous workflow.

2. Name the candidate system precisely

Any model claim must identify the model version and link to the supplied primary documentation for that version. The same applies to context limits, pricing, and API behavior.

No such source is present in this packet, so no model-specific recommendation can be made. Even apparently simple statements about token limits, request formats, or costs would be unsupported here.

3. Create a representative evaluation set

The evaluation set should reflect the defined task and include ordinary cases, difficult cases, and conditions that should trigger refusal or escalation. Scoring criteria must be chosen before reviewing outputs so that the team does not redefine success around convenient examples.

No evaluation data or result is supplied. Therefore, the article cannot claim that RAG, an agent, or a standalone model would meet any quality threshold.

4. Set the human-review boundary

Human review is a product and operational decision, not a decorative safety statement. The team must decide which outputs require review, who can approve or override them, what context reviewers receive, and what happens when review capacity is unavailable.

The boundary should follow the consequence of error. The supplied metadata contains no risk classification or workflow detail, so an appropriate review rate cannot be estimated.

5. Trace data and privacy exposure

A privacy review needs the categories of data entering the system, every service receiving them, retention behavior, access controls, logging, deletion, and any restrictions on secondary use. A general service documentation link is not a substitute for that data-flow analysis.

The current evidence does not establish what data exists or where it would travel. Privacy risk is therefore unknown, not low.

Unit economics cannot be calculated without workload assumptions

A useful estimate separates sourced prices from assumptions and calculations. In symbolic form, a per-period cost model might be written as:

total cost = model usage + retrieval and storage + application infrastructure + human review + failure handling

That expression is a calculation framework, not an estimate. Every term is currently missing either a workload quantity, a unit price, or both.

For model usage, a calculation would need the named model version, sourced input and output prices, expected request count, and measured input and output sizes. Retrieval would need a defined index, query pattern, storage volume, and the applicable sourced prices. Human review would need review frequency and time assumptions. Failure handling would need an operational definition of failure.

The AWS source record does not supply a price, usage tier, region, storage class, request pattern, or data-transfer assumption. It therefore cannot support an S3 cost calculation. Likewise, no model pricing source is supplied.

Any monetary figure added at this stage would be fabricated. The honest status is not estimable from the provided evidence.

A practical go, revise, or stop decision

The review can end in one of three states:

Go to a bounded evaluation

Use this state only after the problem, candidate versions, primary documentation, evaluation set, privacy path, human-review boundary, and cost variables are defined. “Go” should authorize an evaluation, not silently authorize production.

Revise the proposal

Use this state when the problem is credible but the architecture is prematurely specific. The revision should compare at least one ordinary software path with the proposed AI path and identify which requirement makes probabilistic processing necessary.

Stop the proposal

Use this state when the problem can be addressed adequately with simpler software under the agreed criteria, when the required data cannot be handled acceptably, or when evaluation shows that the candidate system misses a critical threshold.

For the supplied packet, the current decision is revise before evaluation. The missing evidence is too fundamental for a technical comparison, but the absence of evidence alone does not prove that the underlying idea should be abandoned.

What would invalidate this recommendation?

This recommendation would change if additional supplied primary sources and project evidence established:

• a specific user or business problem;
• why deterministic software is insufficient under explicit criteria;
• a named model and version with primary documentation;
• version-specific capability, context, pricing, and API evidence;
• a representative evaluation method and acceptance thresholds;
• a defined human-review and escalation boundary;
• a documented data flow and privacy assessment; and
• workload assumptions sufficient for a reproducible cost calculation.

If those materials were provided, the architecture could be reassessed. Depending on the evidence, that reassessment could support an AI component, a conventional software design, or a mixed system.

Make evidence quality part of architecture quality

The important outcome is not a more elaborate diagram. It is a decision that another reviewer can trace from problem to evidence, from evidence to evaluation, and from evaluation to operational limits.

On the current record, approving an AI, RAG, or agent architecture would require unsupported assumptions about capabilities, costs, behavior, privacy, and value. Pausing approval is therefore the practical choice.

If you need a second opinion on an AI proposal, I can help structure a technical audit around its evidence, evaluation plan, delivery risks, and business constraints—without forcing AI into a problem that ordinary software can solve.

Official sources

• GOV.UK
• Amazon Simple Storage Service documentation