A state health department I advised — call it MidCoast Public Health — built a readmission-prediction model that, in pilot, cut 30-day returns by 12 percent. The machine-learning pipeline was sound. Eighteen months of work, validated on three years of claims data. Six months after the model went live, the deputy director for operations pulled the plug because one physician complained the risk scores “looked off,” and nobody inside the agency possessed the explicit authority to audit the algorithm, explain its output, or override it. Multiply MidCoast by every enterprise that has purchased AI without purchasing the governance to go with it. Surveys across sectors place the proportion of AI projects achieving sustained, measurable business value somewhere between 25 and 35 percent. The rest aren’t defeated by technology. They die inside organizational wiring.

Seventy-seven percent of organizations report serious difficulty building AI governance that spans the enterprise, according to the 2023 IBM Global AI Adoption Index. That’s the base rate. The exception — the cohort that makes AI stick — does one thing differently: they assign an accountable owner. A person or committee that can say “pause” and mean it, that reviews model behavior before launch and monitors it afterward, and that has a direct line to the CEO or board. Companies with such structures deploy AI twice as fast and suffer far fewer regulatory scrapes, because the governance board resolves trade-offs before the algorithm hits a live transaction. What follows is not a policy wish list. It’s the gap between ambition and execution, and how to close it.

The Governance Gap

AI governance means the structures, processes, and people who decide what models get built, what data they train on, who can use them, and — critically — who can shut them off. It’s the accountability layer most organizations skip. A 2023 review from the U.S. Government Accountability Office found that more than a dozen federal agencies had deployed AI systems without documenting which officials were responsible for outcomes, creating an audit trail that was “fragmented at best.” The private sector isn’t much tidier: a 2022 McKinsey survey showed that only 18 percent of companies had an enterprise-wide AI risk framework. Governance isn’t a compliance checkbox. It’s the difference between a tool that improves decisions and one that lands an agency on the front page for the wrong reasons.

Consider the case of a large California public agency — I’ll call it WestCounty Human Services — that introduced a fraud-detection algorithm for benefit claims. The data scientists flagged that the training data skewed toward past investigations, which overrepresented certain neighborhoods. The program director moved forward anyway, citing a deadline. Fraud-flag rates in those neighborhoods quadrupled within two months, triggering a civil-rights inquiry. There was no board with the authority to stop the launch. The algorithm was later shelved, but only after the damage was done. When a governance body exists, the same bias flag becomes a checklist item, not a buried email.

The Data Plumbing Nobody Built

Governance fails before a single line of code when organizations don’t know what data they hold, where it came from, or who can alter it. A 2024 survey by Immuta found that 84 percent of data professionals cite data access and quality as a top barrier to AI deployment. Data catalogs remain a novelty outside tech-native firms. The result is models trained on fractured, stale, or biased information — and nobody empowered to fix the source.

Take a Midwestern insurer that built an underwriting algorithm on claims data from 2017-2019, excluding pandemic-era volatility. When the model was deployed in 2022, it penalized groups whose claim patterns had shifted dramatically, because the data pipeline never included a freshness check. A governance framework would have required a data steward to certify the training set every quarter. The insurer had no such role. The model sat in production for eleven months before an external audit flagged the drift. The eventual remediation cost more than the original project.

When Algorithms Escape the Lab

Bias doesn’t stay in the training set. It flows into decisions about loans, hiring, policing, and healthcare. In 2019, researchers showed that a widely used commercial algorithm was systematically assigning lower risk scores to Black patients than White patients with the same health conditions, because the algorithm used healthcare spending as a proxy for need — and inequality shaped the spending data. The study, published in Science, sparked congressional hearings but illustrates the governance void: the company had no standing review board to examine fairness before deployment.

ProPublica’s investigation of the COMPAS recidivism algorithm uncovered similar dynamics a decade ago. The tool was used in sentencing decisions across the country, yet judges rarely knew how the risk scores were derived or tested. Over a hundred jurisdictions lacked any formal process to audit the algorithm’s results. After the ProPublica series, some courts created oversight committees; most didn’t. The technology moved faster than the governance, and people served longer sentences as a result.

What This Means for Your Organization

Governance is not a legal department afterthought. The most effective structures start with three elements: a named accountable executive, a cross-functional review board, and a living model inventory that logs every algorithm in use and its performance. Establish these before you purchase another AI tool. The executive — often a Chief AI Officer or a senior Vice President with operations and ethics authority — owns the decision chain from pilot to decommissioning. The review board, typically comprised of legal, IT, business unit, and external experts, meets monthly to examine new proposals and review drifting models.

Monitoring must be continuous. Set thresholds for accuracy drops, bias indicators, and data drift, and automate alerts. The board’s mandate should include suspending or rolling back a model when those thresholds trip. A White House Blueprint for an AI Bill of Rights recommends that automated systems be accompanied “by meaningful human consideration.” That’s not a technology feature; it’s a governance mandate.

The European Union’s AI Act, phased in through 2026, will require high-risk systems to maintain technical documentation and undergo conformity assessments. Even if your organization operates mostly in the U.S., global supply chains and customer expectations are converging on similar standards. Waiting for regulation to force action is the most expensive route.

One Concrete Step Before You Launch Anything

Before you greenlight your next AI pilot, map the decision chain for model override on a single sheet of paper. Identify who can say “no” at each stage — data selection, training, validation, pre-launch review, post-launch monitoring — and get that person’s signature. If any link is empty, do not launch. This exercise takes two hours. It’s the cheapest insurance you’ll buy.

Organizations that complete this mapping before deployment see fewer fire drills and faster resolution when a model misbehaves. The governance gap is the dullest, least glamorous part of AI transformation. It’s also the part that determines whether your investment becomes a productivity win or a liability. Start with the sheet of paper. Then build the board.

Photo by Igor Omilaev on Unsplash