Why the gap between AI ambition and AI reality in the public sector is closing — and what separates departments that are building genuine capability from those still running the same pilot for the third year running.
Government's relationship with artificial intelligence has matured significantly in the past three years. The question is no longer whether AI has a role in public sector operations — that debate is settled. The question is why so few organisations have moved beyond the pilot stage, and what distinguishes the departments that are building genuine capability from those that are not.
The UK government's AI strategy commits to making the UK a global AI superpower, and significant investment has followed. The National AI Strategy, the Central Digital and Data Office priorities and the creation of the AI Safety Institute all reflect a genuine policy commitment to AI adoption in public services.
And yet, across Whitehall and the broader public sector, the gap between stated ambition and operational reality remains wide. Gartner research has consistently found that the majority of AI projects in large organisations fail to move from pilot to production — a pattern the public sector exemplifies rather than defies. The reasons are structural, not technical.
Gartner Hype Cycle for AI, 2024
The single most common reason government AI pilots fail to scale is data. Not algorithm quality. Not compute availability. Data. Legacy systems built across different decades, by different suppliers, to different standards, hold information that is inconsistent, incomplete and inaccessible. A case management AI that needs to read twenty years of unstructured records held across three incompatible systems is not a three-month proof of concept — it is a multi-year data infrastructure programme with an AI application on top.
PwC's public sector technology practice has estimated that data quality issues account for the majority of AI implementation failures in government. The implication is straightforward: departments that invest in AI before fixing their data foundations are building on sand. The organisations making progress are those that treat data infrastructure as a first-order strategic investment, not a technical footnote.
"The most common failure mode in government AI is not the algorithm. It is the assumption that the data is good enough to learn from."
AI development cycles operate in weeks. Government procurement operates in months. The Crown Commercial Service frameworks that govern most government technology purchases were not designed for the iterative, experimental nature of AI development. By the time a competitive tender is complete, evaluated and contracted, the technology landscape has frequently moved on, the internal team has turned over and the original problem statement has been superseded by a new ministerial priority.
The Procurement Act 2023 introduces flexibility that did not previously exist — including provisions for competitive flexible procedures that better accommodate innovation. Departments that understand how to use these provisions have a genuine structural advantage in building AI capability at pace. Those that default to traditional procurement approaches will continue to find that the process outlasts the ambition.
KPMG's research on digital transformation in the public sector distinguishes between departments that have deployed AI technology and departments that have built AI capability. The distinction matters enormously. Technology can be purchased; capability must be built. Capability means having the data infrastructure to sustain AI in production, the technical skills to maintain and evolve it, the operational processes redesigned around it, and the organisational culture that treats data-driven decision-making as normal rather than exceptional.
The departments making the most progress share a common characteristic: they have senior responsible owners who understand the difference between capability and technology, and who govern their AI programmes accordingly. They are not asking 'how do we deploy this tool?' They are asking 'what needs to be true about our organisation for AI to work sustainably here?'
PwC: The Economic Impact of AI on UK Public Services, 2024
Every significant AI deployment in government involves substantial change to how people work. Case managers whose processes are automated must learn new roles. Policy analysts whose research is augmented by AI must develop new critical skills. Procurement officers whose decisions are informed by AI recommendations must understand how to challenge and override them. None of this happens without deliberate, structured change management.
Prosci's research — now spanning over thirty years and thousands of change programmes — consistently finds that projects with excellent change management are six times more likely to meet their objectives than those without it. In government AI programmes, where workforce concerns about automation are particularly acute, the change management investment is not optional. It is the difference between a system that is used and one that is not.
"Projects with excellent change management are six times more likely to meet objectives than those without it." — Prosci Best Practices Research, 2023
The structural barriers to AI adoption in government are real but surmountable. Data infrastructure requires investment and time but is tractable. Procurement reform is underway. Change management capability exists and can be applied. The organisations that will build durable AI capability are those that approach the challenge with strategic honesty — not asking how to deploy AI quickly, but how to build the conditions in which AI can work sustainably.
For central government, arms-length bodies and the wider public sector, the question is not whether to invest in AI. That question has been answered. The question is whether to invest in the foundations that make AI work — data, people, process and governance — or to continue generating pilots that impress and then disappoint.
This is an area of deep focus for Cairn Novaris's Government & Public Sector and Technology practices. Our cleared practitioners bring direct experience of the environments, constraints and decision-making processes that shape AI adoption in government.
Thinking from practitioners — direct to your inbox. No noise, no sales pitch.