Written by: Andrew Lau (CEO of Think Codex)
As companies invest more in Learning and Development (L&D) programs to help employees acquire new skills and stay competitive, measuring the success of these programs becomes increasingly important. In this article, we'll explore different methodology models that L&D professionals can implement and the critical metrics they need to measure to demonstrate the value of their programs. We start off by examining the most widely used learning effectiveness model – Kirkpatrick Model and then comparing other models against it.
Kirkpatrick Model by Donald Kirkpatrick
The Kirkpatrick Model is one of the most commonly used methodology models in L&D. Developed by Donald Kirkpatrick in the 1950s, this model has four levels of evaluation:
L1 Reaction: This level measures how learners feel about the training program.
L2 Learning: This level measures how much learners have learned from the program.
L3 Behaviour: This level measures whether learners have changed their behavior due to the program.
L4 Results: This level measures the impact of the program on the organization's results.
The Kirkpatrick Model is a valuable framework for evaluating the effectiveness of L&D programs. However, it's important to note that the levels are hierarchical, and each level builds on the previous one. For example, learners are less likely to learn from the program if they don't react positively to the program. They are less likely to change their behavior if they don't learn from it. And if they don't change their behavior, the program is less likely to positively impact the organization's results.
Comparative Methodology Models
In addition to the Kirkpatrick Model, there are other methodology models that L&D professionals can implement to evaluate their programs:
Phillips Model by Jack Phillips
This model takes a step further by not just measuring Levels 1 to 4 which is similar to the Kirkpatrick Model but has a Level 5 measuring the return on investment (ROI) of the training program. The Level 5 ROI measurement uses data from L2 Learning, L3 Behaviour & L4 Results to create a model for finding out what monetary returns the organization is actually getting back from the training dollars spent.
L1 – L4: Following Kirkpatrick Model
L5: Return on Investment
In the form of calculation,
ROI (%) = (Net benefits of the training program / Total program cost) x 100
In the end, the Level 5 calculation comes up with a percentage (%) of ROI. While this is something that is indeed good to measure, there remain a number of challenges in implementing Level 5. Among them:
1. High cost of effort to measure L5 – it takes extensive data collection and working with a number of stakeholders to get accurate data.
2. Delayed Results – as a lot of effort is required, the results of L5 can often be delayed beyond the expected timelines of evaluation by stakeholders.
3. Accuracy of measurement – while calculating, there are a number of assumptions made to put both tangible and intangible factors in focus. Some of these are subjective and up to interpretations depending on stakeholders. It’s important to note that this challenge appears in L4 of the Kirkpatrick & Phillips model to a lesser extent.
4. Ideal but not pragmatic – while it’s great to have these measurements, very few training programs warrant L5 measurements. Jack Phillips himself noted that only 5-10% of training programs require measurements at L5.
5. Lack of data or cooperation from the customer – sometimes a customer just does not have the data required to make a L5 measurement as not all companies track the data of their operations. In the absence of such data, it is often a painstaking process of creating new data sets which can require additional work for stakeholders – which leads to resistance from them.
Brinkerhoff Model by Robert Brinkerhoff
This model is also known as Brinkerhoff’s Success Case Model (SCM) which emphasizes the importance of post-training support and reinforcement to ensure that learners apply what they have learned. Brinkerhoff states that traditional evaluation models such as Kirkpatrick & Phillips models were wrong to just focus on assessing the scope of a training’s effect. He says that the true measure of business results is from performance. SCM is not concerned about finding the average performance of training participants (Kirkpatrick & Phillips Models), but focuses on looking for extreme examples on both ends of measurements.
As such this model measures the performance of the most successful participants and the least successful participants as use cases to understand how well did a learning program perform. The model looks at contrasts or extremes from both ends by seeking to understand 2 important areas:
In a best-case scenario, how well did a program perform? This is derived from analyzing participants who show the most increase in performance after a training program.
In a worst-case scenario, how badly did the program perform? This is derived from analyzing participants who may show the least or no increase in performance after a training program.
Brikenhorff Model
How this model differs from Kirkpatrick & Phillips models is that it requires a few things:
The development of an Impact Model – the definition of what success looks like.
Creating a Success Case Study.
Creating & implementing surveys that can identify best-case and worst-case scenarios.
Conducting interviews.
Documenting successful cases.
Drawing insights, summarizing conclusions, and providing recommendations to stakeholders.
Kaufman Model by Roger Kaufman
This model is based on Kirkpatrick’s model and seeks to improve on it by looking at measurements at a detailed level (L1a and L1b) and big picture level (L5). It’s broken into 5 levels – though if counted, there are actually 6 levels.
L1a Input: This level covers training materials.
L1b Process: This level covers the delivery of the learning process.
L2 Acquisition (Micro): Individual & small group payoffs. Did the learner “acquire” the learning?
L3 Application (Micro): How well the participants utilize what they have learned on the job.
L4 Organizational Results (Macro): Organizational level payoffs – business results.
L5 Customer / Societal Results (Mega): Customers & Society payoffs – the 2 major stakeholders that an organization serves.
Here are some key differences in the Kaufman Model:
1. Group ‘clients’ into Micro (Individual & Small Group), Macro (Organization), and Mega (Customers & Society). This is found in L2, L3, L4 & L5.
2. Divides Kirkpatrick’s Level 1 (Reaction) into Kaufman L1a (Input) & L1b (Process).
3. Introduces a new Level 5 (Customer & Societal Results) which looks at a much broader picture.
So what are the pros of the Kaufman model? Firstly the split in Kirkpatrick’s L1 (Reaction) breaks down the experience into 2 Kaufman factors – supporting materials (L1a Input) and delivery (L1b Process). This helps to isolate the issues of experience by looking at the quality of materials or the quality of delivery.
What are the cons of the Kaufman model? While aspirational and good from a high-level view, trying to measure customer & societal payoffs (L5) require extensive effort, time, and budget. There’s also a high level of ambiguity as there are too many factors at play when we talk especially about customer & societal impacts. In fact, by measuring at L5, it’s hard to isolate these impact factors. This goes against Kaufman’s original intent in where else the goal in L1a & L1b was to isolate factors.
So on one hand, the goal was to improve on Kirkpatrick’s model by isolation (L1a & L1b), L5 instead went against the same goal of isolation, thus giving conflicting views on the Kaufman Model.
Where does Gamification Fit Into all these Models?
Firstly, what is gamification? Gamification is the use of game phycology & game mechanics in non-game environments while achieving non-game outcomes. Such a use case is Learning & Development.
Coming back to learning evaluation models, at the core – most models measure 4 things:
Experience – What was the experience like?
Learning – What did the person learn?
Behaviour – Is there a change in behavior at work?
Results – What individual, team, or business outcomes can be seen?
Gamification’s methodology itself is not an evaluation model but instead is considered one of the most effective ways to drive impact at the levels of Experience, Learning & Behaviour which ultimately leads to Outcome. Let’s break it down to how gamification does that.
Experience – a great experience comes from the right type of engagement. Gamification and game-based learning use the methodology Creation (Ideation & Experimentation) as opposed to just Consumption (Information Download) found in most learning methodologies.
Learning – the most important learning comes from not just the acquisition of the knowledge but rather the application of the knowledge. The application of knowledge is typically where challenges and problems will appear. As such, it’s critical that application can happen within the learning session itself. Game-based learning methodologies such as Business Simulations (emphasis on Business) allow participants immediate application & feedback. Coupled with debriefs that tie it back to organizational context, learning becomes highly effective.
Behaviour – one of the challenges of getting a learner to change their current behaviors is their current habits. Often what is learned is forgotten or replaced with resistance to old habits. Gamification’s iterative and habit-looping techniques get the learner to learn new & better habits which are then overlaid on top of old habits. When learners are back at work, they then begin to practice what they learn from the new habits.
Results – a changed behavior comes down to 3 factors:
The ability to retain the knowledge learned
The retrieval of new habits
The confidence to try out what was learned.
The confidence factor comes from the retention of knowledge and retrieval of new habits. If a learner can remember what they learned and also act out what they learned, then there is confidence to behave in a new way – which then leads to business results.
How to Choose the Right Model
1. Do you have data in place for measurements? If you know where the data is, then a number of models might work.
2. Do you want to create new data sets for measurements? This can often be a complex and long process.
3. Do you have the cooperation of the different stakeholders? Measurements not only require input from the HR professional but stakeholders from the business. Many often have no
4. Do you understand the different models? Do you know what is measurement required and how success is defined in each of the models and how it will fit into your organization?
5. Does the effort required makes it worth getting the results? In essence, this is a cost-benefit analysis. Is the results worth the trouble of going through the measurements – some of which require extensive effort.
Using Different Models For Different Learning Programmes
Should we use different models to measure different training programs? From an outcome perspective, this is ideal. From a practical perspective, the costs may far outweigh the benefit. Our recommendation is using different models should be determined by:
1. What type of results do you really need?
2. What type of effort are you willing to put in?
3. What type of monetary cost are you willing to invest?
4. What type of support do you have in the organization?
5. What level of expertise do you or your vendor have on the different models?
Why Measure L&D Metrics?
Before delving into the different methods for measuring L&D metrics, it's important to understand why it's necessary. Here are some key reasons:
Measuring L&D metrics helps organizations understand the effectiveness of their training programs and identify areas where improvement is needed.
By measuring the ROI of L&D initiatives, organizations can justify the investment in these programs and secure funding for future initiatives.
Improve employee engagement which can lead to increased productivity and reduced turnover.
Increase employee performance because when they have the skills and knowledge they need to do their jobs well, they are more likely to perform better.
Measuring the effectiveness of L&D programs is crucial to ensure that they are achieving their intended goals. By using key metrics and frameworks such as the Kirkpatrick Model, organizations can get a better understanding of the impact of their L&D programs. Subscribe to Think Codex's content and explore our other resources to learn how to measure their effectiveness and achieve your goals using frameworks like the Kirkpatrick Model.