There is an uncomfortable parallel between the AI industry in 2026 and post-war manufacturing in the 1950s. Both produced at impressive scale. Both relied on end-of-line inspection to catch defects. And both treated quality as someone else’s problem: in the factory, it was the inspector’s job; in the AI pipeline, it is the human reviewer’s.
The manufacturing world took roughly three decades and one quality revolution (starting in Japan, spreading across Europe and the United States) to learn that this approach does not work. The AI industry is on track to learn the same lesson, and it does not need to take that long.
The verification tax
Consider the workflow that has become standard in any serious AI deployment. An agentic system analyses documents, extracts data, synthesises findings, and produces an artefact: a report, a codebase, a dataset, a slide deck. Then a human sits down to verify and validate. This is the V&V step, borrowed from systems engineering, and it is where the entire quality burden now lands on human cognition: reading, cross-checking, hunting for errors the system cannot see.
The cost of this is rarely measured but always felt. Every artefact produced by an AI agent carries an implicit verification tax: the time, attention, and domain expertise a human must spend to confirm the output is correct. As AI systems produce more, faster, the tax does not shrink. It compounds. The human becomes the bottleneck in a pipeline that was supposed to remove bottlenecks.
The problem is not that humans are bad at verification. The problem is structural. The reviewer is validating against a specification that exists only in their head. They carry tacit knowledge about what a correct output looks like: domain conventions, semantic constraints, inter-field dependencies, formatting norms, edge cases learned through years of practice. None of this is written down. None of it is measurable. And none of it is available to the system that produced the artefact in the first place.
You might argue that we could delegate this review to another AI agent. In practice, this merely displaces the problem. An LLM asked to “check if this extraction is correct” will apply its own stochastic judgement, not a deterministic specification. It cannot guarantee coverage of domain-specific rules it was never explicitly given. It is, in current jargon, “LLM-as-judge”: an ad hoc assessor with no formal quality framework, no defined tolerances, and no traceability.
This is still inspection. And as W. Edwards Deming spent his career arguing, from his work with Japanese manufacturers in the 1950s to his later influence on Western industry, inspection does not produce quality. It merely sorts good outputs from bad ones after the damage is done. The defect has already been produced; you are just deciding whether to ship it.
What the quality revolution actually taught us
The transformation that Deming helped set in motion (first in Japan, then adopted globally through lean production, the Toyota Production System, and Six Sigma, originally developed at Motorola) rested on a deceptively simple insight: quality must be built into the process, not verified after the fact. The mechanism for this shift was formalisation. You take what you know about what “good” looks like, and you turn it into things you can measure.
In quality engineering terminology, these are Critical to Quality characteristics (CTQs): measurable properties of the output that matter to the end user, derived from higher-level requirements through a structured decomposition. A CTQ is not a vague aspiration (”the output should be accurate”). It is a specification with a target, an upper tolerance, a lower tolerance, and a measurement method. The diameter of this shaft shall be 25.00mm ± 0.02mm, measured by coordinate measuring machine.
Once you have CTQs, everything else follows. You can monitor the process in real time. You can calculate whether the process can reliably produce within specification. You can anticipate failure modes before they occur. And you can design the process itself to make defects structurally difficult to produce.
This last concept, known as poka-yoke in the Toyota Production System, is particularly instructive. A poka-yoke is a constraint built into the process that prevents errors by design. A USB connector that only fits one way. A car that will not shift into reverse above a certain speed. The principle is not to train the operator to be more careful; it is to make the wrong outcome mechanically impossible, or at least immediately detectable.
In software engineering, this philosophy has mature equivalents: Design by Contract (preconditions, postconditions, invariants), static type systems that catch errors at compile time rather than run time, property-based testing that verifies behavioural invariants across thousands of generated inputs. The underlying principle is the same: formalise what “correct” means, and verify it automatically.
The octopus in the labyrinth
Now transpose this to agentic AI. An LLM executing a complex task (extracting structured data from a document, generating a technical report, producing code from a specification) is, in essence, an octopus in a labyrinth. It is remarkably capable: flexible, fast, able to squeeze through passages that rigid rule-based systems could never navigate. But it has a fundamental limitation. It decides for itself whether it has reached the exit. It has no external reference for “done.” It has no walls telling it “not this way.” It is navigating by feel and by pattern, and when it reaches what looks like an opening, it declares success, even if it is in a dead end.
What we propose is to separate the problem into its constituent parts. The octopus is the agent: the flexible, adaptive intelligence that navigates. The labyrinth walls are the compiled constraints: the assertions, invariants, and V&V rulesets that define what constitutes a valid path. And the definition of done is external to the agent, established during build time, not improvised during run time.
The missing middle layer
What does this look like concretely? Consider the current landscape of quality tools available for AI pipelines.
Structured output validation (JSON Schema, Pydantic models) checks whether the output conforms to a data structure. This is necessary, but it operates at the syntactic level. It is type checking. Verifying that a field called “total_amount” is a number is not the same as verifying that the number equals the sum of line items minus the discount, and that the discount rate matches the contractual terms on file.
Guardrails frameworks (NeMo Guardrails, Guardrails.ai) enforce constraints around safety, format compliance, and topic boundaries. Again, necessary. But these are generic: they know nothing about the specific domain in which the AI is operating.
LLM-as-judge approaches use one model to evaluate another’s output. This introduces stochasticity into the quality process itself. Running the same evaluation twice may yield different results. There is no specification, no tolerance, no traceability.
What is missing is the middle layer: domain-aware, semantically rich, measurable quality characteristics that are formally derived from compiled domain knowledge and enforced inline during execution. Not type checking, but semantic contract enforcement.
To make this concrete: consider a system extracting financial data from fund reports. A compiled quality specification for a single field, the net asset value per share, would define not just that the field must be a number (structured output), but that it must be arithmetically consistent with total NAV divided by shares outstanding, that it must fall within a plausibility bound relative to the prior quarter, that its currency denomination must match the fund reference data, and that any deviation triggers a defined response (flag, escalate, halt). Each of these rules is a domain invariant, compiled from expert knowledge, enforced deterministically at run time.
These are not validation rules that an engineer guesses at. They are quality characteristics systematically derived from domain knowledge. And the methodology for deriving them, ensuring coverage, and maintaining them as the domain evolves, is where the real work lies.
Build time and run time
The approach separates AI pipeline design into two distinct phases, borrowing from both quality engineering and software compiler design:
Build time is when domain knowledge is compiled into executable artefacts: the contract specifications (what must be true about each output, with targets and tolerances), the assertion sets (deterministic checks, cross-field validations, plausibility bounds), and the pipeline topology (which steps execute in what order, with quality gates between them). Build time is where you define the invariants and embed the poka-yokes.
Run time is when the agent executes within this compiled framework. The LLM does what it does best: flexible, adaptive processing of unstructured information. But it operates within constraints. Every intermediate artefact is measured against its compiled specifications. Violations trigger defined responses. The definition of done is not the model’s assessment of its own performance; it is the compiled specification’s assessment of the output.
This is not a new idea in principle. Design by Contract has existed since the 1980s. Static analysis catches entire classes of bugs at compile time. What is new is applying this rigour to the specific challenge of AI-generated artefacts, where the execution engine is stochastic by nature and the output is information rather than a binary.
The missing abstraction
The current tooling for AI quality (structured outputs, guardrails, evaluation benchmarks, LLM-as-judge) addresses symptoms without providing an organising principle. Each tool solves a narrow problem. None of them answers the question: where do the quality specifications come from? How are they derived from domain knowledge? How do you ensure coverage? How do you maintain them as the domain evolves?
This is the gap. Without a systematic methodology for compiling domain knowledge into measurable specifications, AI engineers write ad hoc validation rules based on their best understanding of the domain. Coverage is accidental. Maintenance is manual. The quality system is fragile because its foundations are informal.
With domain knowledge compilation, you have a formal, auditable, maintainable derivation of quality specifications from source knowledge. The compilation step is what makes everything downstream, the runtime enforcement, the monitoring, the continuous improvement, possible and grounded.
The road ahead
We are not suggesting that this is easy. The hardest part is not building the enforcement system; it is defining what to enforce. Eliciting tacit domain knowledge from experts and formalising it into measurable specifications is painstaking work. It requires deep domain understanding, iterative refinement, and the intellectual honesty to admit when a quality characteristic is not yet measurable.
But the alternative, the status quo of human-in-the-loop inspection at the end of an AI pipeline, does not scale. It worked when volumes were low. It breaks when you are processing thousands of documents, generating hundreds of artefacts, extracting millions of data points. The verification tax becomes the dominant cost.
The manufacturing world learned this lesson. The question is whether the AI industry will learn it from history, or insist on learning it the hard way.
I spent fifteen years between two worlds that rarely talk to each other: lean manufacturing and quality engineering on one side, machine learning and generative AI on the other. I hold a Six Sigma Black Belt and have built AI systems for critical industries. This article exists because I kept seeing the same quality failures I had learned to prevent on the factory floor reappearing, almost identically, in AI pipelines. The tools to fix them already exist. They just need translating.
This article reflects work at the intersection of AI systems design and quality engineering. We build systems where LLMs act not as autonomous engines but as compilers: transforming structured knowledge into verified, auditable artefacts. The ideas here draw on practice in critical systems engineering, lean production principles, and the daily reality of making AI outputs trustworthy enough for professional use.
The factory analogy becomes useful when it changes the checks upstream. I would still preserve a final commercial review for decisions that source validation alone cannot authorize.