The most common question I get after delivering an AI upskilling programme is: “How do we know this was worth the money?” It’s a fair question. And honestly, for the first few programmes I ran, I didn’t have a great answer.

The problem is that training ROI has always been hard to measure, and AI training adds extra complexity. The benefits are often indirect, delayed, and distributed across teams in ways that don’t show up in a single metric. But “it’s hard to measure” isn’t an acceptable answer when you’re asking for budget.

So here’s the framework I’ve developed over two years and refined across multiple organisations. It’s not perfect. No measurement framework is. But it gives L&D teams something concrete to put in front of leadership.

The four levels (adapted from Kirkpatrick)

I build on the Kirkpatrick Model, which has been the standard for training evaluation since the 1950s. But I’ve adapted it specifically for AI training because generic application misses important nuances.

Level 1: Reaction — Did they engage?

This is the easiest to measure and the least meaningful. Post-programme surveys, Net Promoter Scores, attendance rates. Did participants show up? Did they say they found it valuable?

Specific metrics:

• Programme completion rate (target: above 85%)
• Net Promoter Score (target: above 40)
• Session attendance rates (target: above 90%)
• Qualitative feedback themes

I track these because they’re a hygiene factor. If people aren’t engaging, nothing else matters. But high satisfaction scores alone don’t prove ROI. I’ve seen programmes with stellar feedback that produced zero behaviour change.

Level 2: Learning — Did they actually learn something?

This is where you start measuring substance. Did participants acquire knowledge and skills they didn’t have before?

For AI training specifically, I test three things:

AI literacy assessment. A 20-question quiz administered before and after the programme covering AI concepts, capabilities, limitations, and ethical considerations. I’m not testing technical depth — I’m testing whether managers can have an informed conversation about AI.

Tool proficiency. Can participants demonstrate practical competence with the AI tools covered in the programme? I use task-based assessments rather than written tests. “Here’s a business problem. Use this AI tool to generate a solution. Explain your approach and evaluate the output.”

Decision-making scenarios. Present participants with realistic scenarios — “Your team proposes using AI for X. What questions would you ask? What risks would you flag?” — and assess the quality of their thinking.

Target: Average improvement of 40% or more on the literacy assessment. 80% of participants demonstrating tool proficiency. Qualitative improvement in scenario responses as judged by facilitators.

Level 3: Behaviour — Are they doing things differently?

This is where ROI measurement gets real and where most organisations give up because it requires effort beyond the programme itself.

You need to track whether participants are actually applying what they learned. This means measurement at 30, 60, and 90 days post-programme.

Specific metrics I track:

• AI tool adoption rates. Are participants using the AI tools they learned about? Track license usage, login frequency, and active usage data. If you trained people on an AI analytics platform and nobody’s logging in after 60 days, the training didn’t stick.
• AI opportunity identification. How many AI use cases have participants proposed since completing the programme? Track proposals submitted to governance boards, innovation teams, or IT.
• Cross-functional collaboration. Are trained managers having more productive conversations with technical teams? I survey both sides — the managers and the data/IT teams they interact with.
• Manager confidence scores. A self-assessment survey asking managers to rate their confidence in evaluating AI proposals, communicating about AI to their teams, and identifying AI opportunities. Administered quarterly.

The Australian HR Institute provides frameworks for measuring behavioural change that complement these AI-specific metrics. Their competency-based assessment approach works well for tracking skill application over time.

Level 4: Results — What’s the business impact?

This is what leadership actually cares about. Did the investment produce measurable business outcomes?

Here’s where you need to be honest about attribution. AI training alone rarely produces a directly measurable financial return. It produces capability, which enables projects, which produce returns. Claiming a straight line from a two-day workshop to revenue growth is intellectually dishonest.

What you can measure:

AI project initiation rate. How many AI projects were proposed or launched by trained managers in the 6-12 months following the programme? Compare this to the rate before the programme and to untrained cohorts.

Time to AI project delivery. Are teams with trained managers delivering AI projects faster? If a trained manager can evaluate a vendor proposal in a week instead of a month, that’s measurable.

AI project success rates. Are projects led by or involving trained managers more likely to meet their objectives? Track project outcomes and correlate with training participation.

Risk incident reduction. Are trained organisations experiencing fewer AI-related incidents — data breaches, bias complaints, compliance issues? This is a defensive metric, but avoiding a single incident can be worth more than the entire training budget.

The calculation

Here’s how I present the financial case. It’s not perfect, but it’s credible.

Direct costs: Programme development, facilitation, participant time, tools, and materials. Calculate fully loaded — including opportunity cost of participant hours.

Measurable benefits:

• Value of AI projects initiated by trained managers (use project-level ROI estimates)
• Time savings from faster AI vendor evaluation and project scoping
• Risk reduction value (estimated cost of avoided incidents multiplied by probability reduction)
• Efficiency gains from AI tool adoption (hours saved multiplied by hourly cost)

Indicative benefits (harder to quantify, still worth noting):

• Improved organisational AI literacy
• Better cross-functional collaboration
• Increased employee engagement and retention among trained cohorts
• Faster decision-making on AI investments

In my experience, the measurable benefits alone typically show a 3-5x return on investment for well-designed programmes. Adding the indicative benefits pushes it higher, but I prefer to be conservative in reporting.

Common mistakes in measuring AI training ROI

Measuring too early. Behaviour change takes time. If you’re surveying participants two weeks after the programme and concluding it didn’t work because nothing’s changed yet, you’re measuring wrong. The meaningful signal comes at 60-90 days minimum.

Ignoring the counterfactual. What would have happened without the training? If your competitors are all adopting AI and your managers can’t engage with it, the cost of not training is enormous but invisible. Frame ROI as both gain from training and cost of inaction.

Focusing only on financial metrics. Some of the most valuable outcomes — better risk management, more informed decision-making, improved collaboration between business and technical teams — don’t reduce to a dollar figure. Present them qualitatively alongside the numbers.

Not having a baseline. If you don’t measure knowledge, behaviour, and outcomes before the programme, you can’t demonstrate change. Always run baseline assessments, even if they’re simple.

Making it practical

You don’t need a data science team to measure training ROI. Here’s the minimum viable measurement stack:

1. Pre and post literacy assessment (Google Forms works fine)
2. Satisfaction survey at programme end
3. Behaviour survey at 30, 60, and 90 days (three short emails)
4. AI tool usage data from IT (login reports)
5. AI project pipeline tracking (a simple spreadsheet)
6. Quarterly review of business outcomes

The total effort: maybe two hours per month of an L&D coordinator’s time, plus the initial assessment design.

The payoff: a credible ROI story that justifies continued investment and demonstrates L&D’s strategic value. That’s worth the effort.