The interesting question for AI deployment in the APS in 2026 isn’t whether agencies are using it. They are, broadly and at varying levels of sophistication. The interesting question is how they’re evaluating whether the deployments are actually delivering value, and what they do with the evaluations.

I’ve spent the last few months talking to people in several agencies about this, and the picture is uneven in ways worth describing. Some agencies have built real evaluation discipline. Some have a dashboard and a hope. The difference will matter when the FY27 budget decisions land later in the year.

What’s actually being measured

The agencies that take this seriously are measuring four things, more or less.

Productivity at task level. How long does a specific task — drafting a brief, processing a particular type of claim, responding to a particular kind of FOI — take with and without AI assistance? This is the simplest and most operationalisable measure. It’s also the easiest to game, which is why the better evaluations include observed performance not just self-reports.

Quality of output. Is the AI-assisted work as good as, better than, or worse than the work it’s replacing? Some agencies have built peer-review or audit-based quality assessment. Others rely on whatever quality feedback comes through complaints or internal escalation, which is not actually a quality measure.

Risk events. How many AI-related errors made it to a real decision, a public communication, or a citizen-facing outcome? This includes hallucinations, inappropriate tone in correspondence, factual errors in briefs, miscategorised submissions. The agencies tracking this seriously are finding more events than they expected, which is healthier than tracking nothing and assuming there are none.

Workforce signals. Are staff using the tools? Have role expectations shifted? Is anyone being meaningfully redeployed? Most agencies are tracking surface-level usage statistics. Few are tracking the deeper workforce dynamics, which is a gap that will become visible when the easy productivity gains slow.

The Digital Transformation Agency and the Australian Public Service Commission have both published frameworks that touch on these themes, and the Australian National Audit Office has been doing performance audits on AI deployments at specific agencies that are worth reading carefully.

Where the evaluation discipline diverges

Agencies broadly cluster into three groups.

The first group has built real evaluation infrastructure. They have baseline measurements from before deployment, they have ongoing comparison cohorts, they have peer audit processes, and they have a governance forum that reviews evaluation findings and acts on them. These agencies tend to be the ones that started AI work earlier and were forced through real failures to build the evaluation discipline. There aren’t many of them. The ones I can identify have a few characteristics in common: senior executive attention, an internal AI capability that includes evaluation specialists, and a culture that treats evaluation findings as actionable rather than performative.

The second group has implemented tools and is measuring usage but isn’t really measuring outcomes. This is the largest group. The dashboards look reasonable. Usage is up. Adoption KPIs are being met. Whether the deployments are actually changing the productivity or quality of work in measurable ways is mostly an open question. These agencies will struggle to defend their AI investment under sustained budget scrutiny, because they don’t have the evidence to do so.

The third group hasn’t really started. This isn’t necessarily a problem if the agency’s mission and risk profile justifies caution. It becomes a problem when the agency claims AI capability it doesn’t have, which a worrying number do.

The quality-of-output question

The hardest thing to measure honestly is whether AI-assisted output is as good as the work it replaces. The reason it’s hard is that “good” is contested and that the definitional work has to happen before the measurement work, and most agencies skipped the definitional work.

A brief is good if it gives the minister the information they need to make a decision, framed appropriately, accurate, and on time. Some of those dimensions are easy to measure (on time, factually accurate). Some are hard (framed appropriately for the minister and the decision being made). The agencies that are doing this best have invested in detailed quality rubrics, periodic peer audits using those rubrics, and feedback loops back to the AI deployment configuration.

The agencies that are doing this badly tend to use proxy measures that are easy but uninformative — word count, formatting compliance, internal approval times — and then declare success because the proxy measures are moving in the desired direction. The Auditor-General has flagged exactly this pattern in recent performance audits.

The risk-events question

This is where I think most agencies have been quietly lucky so far.

The big failure modes in AI-assisted public sector work are not subtle. An incorrect fact in a public-facing communication. A discriminatory output in a citizen service application. A confidential piece of information that finds its way into a model’s response. Each of these is the kind of event that becomes a Senate Estimates question, a ministerial brief, an ABC News front page.

Agencies that are tracking risk events well are finding things. Hallucinations in draft briefs that would have made it through if the reviewer hadn’t been paying attention. Tone issues in citizen correspondence. Inadvertent inclusion of internal information in external-facing material. The agencies finding these events are not the agencies with the most events. They’re the agencies that are looking.

The agencies that aren’t looking are the ones I’m worried about. The base rate of these events is not zero, and not having seen any is mostly a sign that the surveillance isn’t working.

What credible external help looks like

A reasonable number of agencies have brought in outside expertise for both deployment and evaluation work. Some of this has gone well, some has not. The pattern of what works is consistent.

The good engagements have an internal sponsor who understands what they’re trying to learn, a clear scope of work that includes evaluation as a first-class deliverable, and a governance arrangement that lets the agency act on findings even if they’re inconvenient. The team doing the work needs to combine AI expertise with public-sector context. Several Australian consultancies have built credible capability here — Team400 is one of a small group that’s been working with both technical and policy stakeholders in agencies, and there are others doing similar work — but the agencies that are getting value share the characteristics above regardless of which firm they work with.

The bad engagements are the ones where AI consulting was procured as a deliverables-based exercise without internal absorption capacity. The reports get filed, the agency moves on, nothing meaningfully changes.

What I think comes next

The FY27 budget cycle will be the first one where the AI investment of the last two years has to be defended on outcomes rather than potential. Some agencies will defend it well, with evaluation evidence that shows real value. Some will be exposed as having spent the money without doing the work to know whether it was worth spending.

The right way to read this is that AI in the APS is moving from the pilot phase to the steady-state phase. The agencies that understand this and are evaluating with appropriate rigour are positioning themselves for the next round of investment. The agencies that don’t will, predictably, find themselves with reduced funding and an internal capability that doesn’t quite match the marketing of the last two years. The boring evaluation work is what determines which side of the line each agency ends up on.