“I’m pretty good with AI tools.”

This statement from a workshop participant was followed by 45 minutes of struggling with basic prompts. Self-assessment hadn’t matched reality.

The reverse happens too. Another participant dismissed her AI skills as “beginner” before demonstrating sophisticated prompt engineering that outstripped most of the room.

Self-reported AI capability is unreliable. People lack calibration for skills they’re still developing. Yet many organisations base development planning on self-assessment alone.

Better assessment methods exist. Here’s how to evaluate AI capability more accurately.

Why Self-Assessment Fails

Self-assessment has fundamental limitations:

Dunning-Kruger Effect

People with limited skill often overestimate their capability (they don’t know what they don’t know). Those with higher skill sometimes underestimate (they assume others are equally capable).

This well-documented effect makes self-assessment systematically unreliable.

Reference Point Problems

“Good at AI” compared to what? Without clear reference points, people anchor on whatever comparison feels natural—which varies enormously.

Confidence vs. Competence

Some people have high confidence with moderate skill. Others have low confidence despite high skill. Self-assessment captures confidence more than competence.

Social Desirability

People sometimes report what they think they should be or what makes them look good, not actual capability.

Lack of Visibility

People may not recognise advanced capabilities in others, making self-assessment relative to limited observation.

Better Assessment Approaches

Multiple methods provide more accurate assessment:

Task-Based Assessment

Assess capability through actual tasks:

Direct observation:

• Give people AI tasks to complete
• Watch how they approach them
• Assess process and outcomes
• Note capability indicators

Task scenarios:

• Present realistic work scenarios
• Have people complete using AI tools
• Evaluate results against criteria
• Compare across individuals

Time-pressured tasks:

• Speed reveals fluency
• Quick, appropriate use indicates capability
• Hesitation and inefficiency indicate gaps

Task-based assessment shows what people can actually do.

Behavioural Assessment

Observe work behaviour:

Manager observation:

• How does the person use AI in regular work?
• What quality do they produce?
• How efficiently do they work?
• What questions or struggles occur?

Peer feedback:

• Who do colleagues go to for AI help?
• What reputation does the person have?
• What do peers observe about capability?

Work output review:

• Assess AI-assisted work products
• Evaluate quality, appropriateness, efficiency
• Look for indicators of sophisticated vs. basic use

Behavioural evidence grounds assessment in reality.

Knowledge Assessment

Test understanding that underlies skill:

Conceptual knowledge:

• Understanding of AI capabilities and limitations
• Awareness of appropriate use cases
• Grasp of verification requirements
• Ethics and policy knowledge

Procedural knowledge:

• How to accomplish common AI tasks
• Troubleshooting approaches
• Workflow integration
• Tool features and functions

Situational judgment:

• Given scenario X, what’s the appropriate AI approach?
• When should AI be used vs. avoided?
• What verification is needed for this output?

Knowledge supports skill but doesn’t guarantee it.

Calibrated Self-Assessment

If using self-assessment, improve its quality:

Specific anchors:

• Define what “beginner,” “intermediate,” and “advanced” mean specifically
• Provide behavioural descriptions for each level
• Give examples of what capability looks like at each level

Comparative questions:

• Rather than “how good are you,” ask “how often do you do X”
• Frequency questions are more accurate than ability questions
• Specific behaviours are easier to assess than general capability

Multiple dimensions:

• Assess different capabilities separately
• Someone might be strong in content generation but weak in data analysis
• Disaggregated assessment is more accurate

Triangulation:

• Combine self-assessment with other methods
• Use self-assessment as one input, not the sole input

A Capability Assessment Framework

Organise assessment around key capability dimensions:

AI Tool Proficiency

Can the person use AI tools effectively?

Indicators:

• Smooth tool navigation
• Appropriate feature utilisation
• Efficient workflows
• Ability to use advanced features when valuable

Assessment methods:

• Task completion
• Tool usage observation
• Feature knowledge check

Prompt Engineering

Can the person communicate effectively with AI?

Indicators:

• Clear, specific prompts
• Iterative refinement
• Context provision
• Output format specification

Assessment methods:

• Prompt review
• Iterative task completion
• Prompt writing exercises

Output Evaluation

Can the person assess AI outputs critically?

Indicators:

• Error identification
• Fact checking behaviour
• Quality judgment
• Bias recognition

Assessment methods:

• Evaluate AI outputs with embedded errors
• Observation of verification behaviour
• Quality assessment of AI-assisted work

Appropriate Application

Does the person know when and how to apply AI?

Indicators:

• Selects appropriate tasks for AI
• Recognises where AI helps and hurts
• Integrates AI into suitable workflows
• Avoids inappropriate uses

Assessment methods:

• Scenario-based judgment questions
• Observation of real work decisions
• Task selection exercises

Integration and Efficiency

Can the person integrate AI into productive workflows?

Indicators:

• Smooth workflow integration
• Time efficiency with AI
• Reduced friction in AI use
• Sustainable AI practices

Assessment methods:

• Workflow observation
• Productivity comparison
• Time studies

Assessment for Different Purposes

Match assessment approach to purpose:

For Individual Development Planning

• Detailed capability profile across dimensions
• Specific gap identification
• Comparative (to role requirements, not other people)
• Collaborative—person participates in interpretation

For Training Program Design

• Aggregate across population
• Distribution by capability level
• Segment analysis (by role, location, etc.)
• Gap analysis relative to requirements

For Hiring

• Standardised across candidates
• Job-relevant task demonstration
• Comparative across candidates
• Predictive of job performance

For Team Formation

• Complementary capability identification
• Relative strengths within team
• Gap identification at team level
• Role-based assessment

Purpose shapes methodology.

Assessment Pitfalls to Avoid

Common mistakes in AI capability assessment:

Single-method reliance: Don’t rely on any single method. Triangulate across methods.

Static assessment: Capability changes. Assess periodically, not once.

Tool-specific focus: Capability with one tool may not transfer. Assess transferable skills, not just tool knowledge.

Binary thinking: Capability isn’t binary. Assess degrees and dimensions.

Ignoring context: Same person may perform differently in different contexts. Consider environmental factors.

Forgetting purpose: Assessment for development differs from assessment for selection. Match methods to purpose.

Building Assessment Capability

Developing good AI capability assessment requires investment:

Assessment Design

• Determine dimensions to assess
• Develop assessment instruments
• Create scoring criteria
• Establish reference points

Assessor Development

• Train managers to observe effectively
• Develop consistent evaluation standards
• Build calibration across assessors
• Create quality assurance processes

Infrastructure

• Technology for assessment delivery
• Data management for results
• Reporting and analysis tools
• Integration with development systems

Continuous Improvement

• Validate assessment against outcomes
• Refine instruments based on experience
• Update for evolving AI capabilities
• Incorporate feedback

Assessment capability is itself a capability worth developing.

From Assessment to Action

Assessment without action is waste. Connect assessment to development:

Individual Development Plans

Assessment reveals gaps. Development plans address them:

• Prioritised development areas
• Specific learning activities
• Timeline and milestones
• Support and resources

Programme Design

Aggregate assessment informs programme choices:

• Content priorities
• Level targeting
• Differentiation needs
• Resource allocation

Progress Tracking

Reassessment shows progress:

• Capability development over time
• Programme effectiveness
• Remaining gaps
• Next development priorities

Assessment, development, reassessment—a continuous cycle.

Know where your people actually are with AI capability. Not where they think they are. Not where you hope they are.

Accurate assessment enables effective development.

Start assessing properly. Watch development accelerate.