Why AI pilots stall between proof of concept and production

By Lexi · 2026-09-30

Pilots stall because post-demo work is engineering and governance, not modeling. We map the failure modes and concrete fixes using NIST, Google, and McKinsey.

Pilots stall because the work after the demo is engineering, governance, and operations, not modeling.

What actually breaks after the demo?

Pilots break when ungoverned data, configuration sprawl, and missing runtime controls collide with production change and accountability. Google’s Hidden Technical Debt paper describes the mechanism plainly: data dependencies, configuration complexity, and entanglement create brittle systems that pass a demo then fail under routine changes in schemas, features, and consumers. NIST’s AI RMF 1.0 adds the governance dimension: if we cannot show that a system is valid and reliable, secure and resilient, explainable and interpretable, privacy-enhancing, fair, and accountable, the deployment stalls. The McKinsey work on gen AI pilots reports that fragmentation across platforms and immature orchestration adds cost and slows integration. Put simply, the demo optimizes for a single use on a frozen dataset. Production is continuous change with consequences.

Which debts hit first when we try to scale?

The first debts that hit are data dependencies we did not declare, configuration we did not version, and boundaries we did not enforce. Google’s Hidden Technical Debt lists specific traps we see in the field:

• Data dependencies: upstream teams change a column name or distribution, our pilot has no validation, and predictions degrade quietly.
• Configuration complexity: feature flags, thresholds, and preprocessing live in notebooks, not in versioned configs. Reproducing the pilot is slow or impossible.
• Entanglement: one model serves multiple use cases. A change for one stakeholder breaks another.
• Boundary erosion: the ML system reaches into systems of record directly. We cannot isolate failures or roll back safely.
• Undeclared consumers: shadow consumers read our outputs. A fix surprises downstream processes.
• Feedback loops: the model influences the data it later trains on. Without monitoring, we amplify bias or lock in errors. These show up before we touch scale. They show up as incidents, regressions, and expensive rewrites when we leave the lab. The survey of deployment case studies by Paleyes, Urma, and Lawrence reinforces this. Practitioners faced issues at every stage, from data collection and versioning to monitoring and organizational alignment. None of this is solved by a better model architecture.

How do we prove trustworthiness without slowing to a crawl?

We prove trustworthiness by turning NIST AI RMF goals into concrete, automatable controls tied to owners and runbooks. NIST’s categories are practical when translated:

• Valid and reliable: we define performance metrics for the business context, establish acceptance thresholds, and test for training and serving skew. Google’s ML Test Score points to specific tests for skew and decay. We wire these to alerts and approvals.
• Secure and resilient: we isolate runtimes, sign artifacts, enforce environment parity, and plan graceful degradation. McKinsey’s guidance on MLOps engineering work, like pipeline hardening and incident management, fits here.
• Explainable and interpretable: we document input features, model limitations, and rationale. We provide summaries or example-based explanations suitable for the domain. We establish recourse paths if a decision needs review.
• Privacy-enhancing: we minimize data, mask sensitive fields, and track provenance. We write explicit data-use policies into the pipeline.
• Fair: we specify slices to monitor and evaluate. We include fairness checks in test and monitor suites, tied to the business risk owner.
• Accountable: we assign decision ownership, define escalation paths, and log decisions with traceability to data and model versions. We do not bolt a governance portal on top of a brittle stack. We build controls into the pipeline that already delivers value. We measure only what we can act on. We route exceptions to humans early rather than late.

What minimum platform do we need to get out of pilot purgatory?

We need a thin, shared platform that standardizes the boring parts and leaves model choice flexible. McKinsey’s advice to consolidate on a common platform and mature orchestration matches our experience and the ML Test Score rubric. At minimum, we standardize:

• Data contracts and validation at ingestion: schema, types, ranges, and statistical checks enforced before training and serving.
• Versioning and lineage: data snapshots, feature sets, training code, model artifacts, and configs tracked and reproducible.
• Environment parity: the same containers and dependency locks in dev, staging, and prod.
• Deployment safety rails: canary releases, guardrail metrics, automated rollback, and feature flags.
• Monitoring and alerting: performance, data drift, feature health, latency, and cost, with thresholds and paging policies.
• Incident management: on-call rotation, runbooks, and post-incident reviews focused on systemic fixes.
• Access and change control: role-based access, approvals for model promotion, and audit logs. McKinsey’s MLOps guidance maps the engineering tasks after a pilot to these platform capabilities. The ML Test Score gives us a checklist that we can score and track.

How should we prune the portfolio and focus resources?

We should cut pilots that cannot express a business owner, a data contract, and a control plan. McKinsey’s report on the post-honeymoon phase says pilots stall when resources spread thin and platforms fragment. We agree. We ask three questions per pilot:

• Who is the accountable business owner for the decision the system influences, and what metric will they own after go-live?
• What is the explicit data contract, and who maintains the upstream systems? If no contract exists or cannot be enforced, we stop.
• Which NIST AI RMF controls are material for this use case, and which tests and monitors will satisfy them? If we cannot map controls to automatable checks, we stop or redesign. This reduces the number of active pilots, but increases the number that graduate. It also forces early coordination with risk and product. McKinsey recommends integrating risk, business, and product leadership early for exactly this reason.

What tests keep models from drifting into the ditch?

We keep models healthy with a small set of tests and monitors that catch the common failures. Google’s ML Test Score proposes concrete tests across data, model, and infrastructure. We prioritize:

• Data validation tests: schema, ranges, categorical domains, and missingness, failing fast on violations.
• Training and serving skew checks: compare distributions between training data and live traffic.
• Canary tests: deploy to a small shard, watch guardrail metrics, and promote only on pass.
• Rollback tests: dry-run rollbacks and verify state integrity.
• Drift monitors: population drift and performance decay, with retraining or human review triggers.
• Dependency checks: alert on version changes in key libraries or upstream APIs. These are not optional. The ML Test Score frames them as production readiness, not nice-to-have. NIST’s focus on validity and reliability makes the link to governance clear. We add runbooks so alerts lead to action, not dashboard noise.

Where do risk controls meet the build pipeline?

Risk controls meet the build pipeline at the points where data, code, and models change. We wire NIST AI RMF controls to specific lifecycle stages:

• Scoping: document intended use, impacted decisions, and harm scenarios. NIST emphasizes this up front. We gate funding on this doc.
• Data management: enforce contracts, document lineage and consent constraints. We fail builds on violations.
• Model development: include explainability notes and known limitations as part of the artifact. We require tests for slices tied to fairness goals.
• Deployment: canary, approvals, rollback, and change records. We tie this to accountable owners.
• Monitoring: drift, performance, and incident logging. We hold post-incident reviews focused on prevention.
• Incident response: defined triggers and escalation for outages or harm events. We test this like a fire drill. This keeps risk integrated without creating parallel paper processes that slow teams. It also gives risk teams real signals instead of slideware.

What does the failure-to-control mapping look like in practice?

Here is how common failure modes map to practical controls, with sources backing each control. We implement these as platform defaults.

How do we size the effort and timeline without made-up numbers?

The effort depends on current platform maturity, data quality, and regulatory exposure. As a range, the core controls above can be stood up in weeks if you already have containerization, CI, and observability, and in months if you are building those foundations. The long pole is usually data contracts, since they require coordination with upstream teams and often schema refactors. Regulated contexts add documentation and review cycles. The mechanism is constant: build the thin platform, move one pilot through it end to end, then scale.

What operating model keeps this running after go-live?

We formalize a joint ownership model across business, engineering, and risk. McKinsey’s guidance to integrate these roles early holds after go-live too. We implement:

• A product owner in the business who owns outcome metrics.
• An engineering team that owns the platform and SLOs for latency, availability, and model promotion.
• A risk partner who signs off on NIST control coverage and reviews incidents.
• A cadence for reviews: weekly operational, monthly performance and drift, quarterly control audit.
• A simple intake that requires the three gating answers noted earlier. The survey by Paleyes and colleagues highlights organizational misalignment as a persistent barrier. This model addresses that with named owners and routine.

What does a worked example look like?

Take invoice coding in Accounts Payable. The pilot reads a hundred invoices from one vendor set, predicts GL codes with high accuracy, and impresses stakeholders. Then it stalls. Here is how we move it through production using the sources above as the design spine.

1. Scope and risk profile. We document intended use and failure modes per NIST’s AI RMF. Misclassification leads to mispostings and rework. We set a business metric, percent of invoices auto-coded with less than a defined correction rate, owned by the AP lead.
2. Data contract. We define a schema for invoice inputs: vendor_id, line_item_text, unit_price, currency, tax_code, and attachment OCR text. We write range checks and categorical domains. We identify the upstream owners of ERP extracts and OCR service. We enforce validation at ingress. This addresses the data dependency debt highlighted in Hidden Technical Debt and aligns with ML Test Score data validation.
3. Single feature pipeline. We implement tokenization, currency normalization, and vendor history joins in a library used in both training and serving. We test for training and serving skew in CI. This comes straight from the ML Test Score and McKinsey’s call for environment parity.
4. Versioning and lineage. We snapshot the training dataset, record feature versions, pin dependencies, and log the model artifact with a manifest including explainability notes and limitations. This supports NIST’s validity, explainability, and accountability goals.
5. Deployment safety. We canary the model on 5 to 10 percent of live invoices while the rest follow the old rules engine. We set guardrails: correction rate, exception queue size, and latency. If any guardrail breaks, we auto-roll back. This follows the ML Test Score guidance on canary and rollback and McKinsey’s incident focus.
6. Monitoring and alerts. We track input drift by vendor, document type, and amount bracket. We monitor correction rates by slice to watch fairness across vendors or regions as appropriate to the business. We alert the AP lead and engineering if drift or correction rates cross thresholds. This ties to NIST’s fairness and reliability focus.
7. Human in the loop and recourse. Exceptions route to AP analysts with model suggestions and rationale, which analysts can accept or correct. We record corrections to feed retraining. NIST’s accountability and explainability requirements show up here in workflow, not a slide.
8. Incident response. We define an on-call engineer, a playbook for OCR outages, and an escalation to the AP lead. We rehearse a rollback. McKinsey’s MLOps piece stresses incident management, and the ML Test Score calls for rollback drills.
9. Review cadence. Weekly operational review tracks guardrails and backlog. Monthly review checks drift and slice performance. Quarterly review audits control coverage against NIST categories. This example keeps technology aligned to the operational reality. It shows how the sources translate to day-to-day actions that move a pilot to production and keep it there.

What should we do this quarter?

We would do four things:

• Pick one pilot with a clear business owner and move it end to end through the controls above. Prove the path once.
• Stand up the thin platform: data validation, versioning, environment parity, canary, rollback, and monitoring. Use the ML Test Score as the checklist.
• Write down the NIST AI RMF mapping for that pilot, with owners and runbooks. Keep it to one page per category.
• Kill or pause pilots without a data contract or a control plan. McKinsey recommends this consolidation to free resources. Pilots stall when we treat production like research. They move when we treat production like operations. The sources agree on the mechanism. NIST defines what trustworthy means in production. Google’s work explains why debt kills promising demos. McKinsey describes the platform and portfolio moves that unblock scale. Our job is to implement the boring parts and wire them to owners. The rest, including better models, can follow.