How to build a training scorecard that will help to separate time-spending on infotainment and highly useful performance-shifting training.
Training and workshops have become a standard part of learning process in any organization. In my opinion, the most important challenge for the organizations is to be able to detect the difference between less useful infotainment and highly useful performance-shifting training. In this article, we will discuss how to estimate the effectiveness of the training and build a training scorecard.
- Kirkpatrick Levels Model
- The Cost of the Measurement
- Training Scorecard Template: Preparation
- Level 1: Reaction, Level 2: Learning, Level 3: Behavior, Level 4 – Impact
- KPIs for eLearning
The Problem of Training Measurement
Imagine a workshop that is delivered by a charismatic trainer, who knows the topic and easily engages with trainees.
Sounds good, but we still need to ask a number of questions:
- Did this training give our team required knowledge and skills?
- Were those new skills actually applied?
- How do we know that this training will finally lead to performance improvements?
It’s also a good idea to know:
- What parts of the training worked well and what parts need to be improved?
- Should we repeat the training on this topic?
- Should we hire this trainer again or find someone new?
These are the challenges that CEO/HR faces.
Trainers in their turn, need a data-powered feedback loop that will help them to improve their training products.
Trainers need to understand how to evolve the training program in order to increase the outcomes and the impact on the organization’s performance.
Kirkpatrick Levels Model
One-time measurement won’t provide us with the objective results. For effective evaluation of the training, we will need to have several control points separated in time periods.
The Kirkpatrick levels framework suggests reviewing a training from 4 perspectives:
After some time we can track:
- Level 3. Behavior (knowledge/skills applied in practice)
- Level 4. Impact (impact on ultimate business performance)
If the training is conducted by a third party expert, then we also need to take into account a shortlisting process. There should be some prerequisites for trainers and training:
- How qualified is the trainer, and
- How the suggested training is aligned with the company’s strategy.
The Cost of the Measurement
If we put these four steps on a chart with “The cost of measurement” on one axis and “The benefit of measurement” on another, we’ll see that:
- The easiest to measure is the immediate reaction (Level 1) after the training and training takeaways (Level 2).
- Measuring prerequisites requires some quantification of a screening process, but it is still doable.
- While the levels Level 3 and Level 4 are the most interesting from the viewpoint of possible findings, the cost of the measurement increases.
Training Scorecard Template
Let’s discuss a template that one can use to evaluate training effectiveness in their organization. I’ll use BSC Designer software to demonstrate how this process can be automated.
My recommendation is to start with a good strategy map that links your financial, customer-related, and internal goals with your learning initiatives. This will give a direction to all of the following management efforts.
The next step is to measure prerequisites. To do this, we will need to quantify the following data:
- Trainer qualification
- Trainer communication skills
- Estimated training costs (travel expenses, accommodation, trainer fee)
- The match between training program and strategic goals
When quantifying these parameters, make sure that the estimation is not done by the head of HR alone, ask key members of your team to share their opinions.
When to measure? A few weeks before the training.
The software will help to:
- Assign weight to each criterion
- Store individual evaluations and the weights of the parameters
- Calculate the score for each of the candidates
Level 1. Reaction
At this level, we need to measure after-training feedback that will help to get an idea about how engaging this training was. These questions could give some data for the indicators:
- How do you feel about the training? (negative, neutral, positive)
- Would you recommend this training to your colleagues?
- Do you want to repeat this training in the future?
The list might go on, as more indications of high engagement can be found:
- The number of questions asked during and after the training
- Subjective estimation of the engagement of the audience (were people participating or checking emails on their smartphones?)
The software helps to:
- Collect the evaluation for each indicator shared by each participant
- Give natural language choice options
- Calculate the average scores
Measurement date: on the date of the training.
Level 2. Learning
The easiest way to quantify learning outcomes is to conduct a final test that will show how well the students understood the material.
The metrics could be:
- Exam pass rate, %
- Average exam score, %
- Before/after training test results
The learning outcomes can be tested in different ways. For example:
Instead of using a simple test, one can ask trainees to practice something that was learned and evaluate the actual behaviour.
Respectively, the practice-related indicators will need to be implemented.
In this way, we are not only testing but helping students to establish new behavior patterns.
Measurement date: at the end of the training.
We did the required minimum. The cost of the measurement for the next levels will increase. Should we go there?
If we stop our measurement on this level, then we are at risk of “false positive” situations, when a charismatic trainer delivers an engaging talk that is well-accepted by an audience (a kind of infotainment) but will rarely result in performance improvements in the organization.
Level 3. Behavior
Conducting tests is a good starting point to make sure that the trainees have obtained required knowledge, but what we actually want to see is a certain change in their behavior patterns. The quantification, in this case, depends on the nature of the process.
In some cases, the indicators of new behavior will be as specific as:
- Time reduction achieved, %. If the goal of the training was to do something more effectively or simply faster, then you might track it by a KPI.
- Cost reduction achieved, %. If the goal of the training was to do something in a more effective way, then you might track direct cost reduction, which is the result of the training.
- Increased quality (we discussed before the way to quantify quality) or more specific Reduce of the return problems rate, %. The KPI is applicable where the problem is clearly defined and linked to the final result.
By the way, a repeated problem is one of the negative drivers of customer loyalty. If you can decrease this number, you’ll make your customers happier.
- Performance improvement, %. The KPI must be linked to a specific performance index. For salespeople, it might be the number of sales generated, for a copywriter, it might be readers to leads conversion. It’s possible to find performance metrics even for creative teams, like graphic designers.
In most cases, we will need to spend time finding good indicators that would quantify new behavior patterns. Our KPI system will help to frame the brainstorming process.
It is a good idea to ask the trainer beforehand about expected changes in the behavior and the best way to achieve them in the organization.
- For example, one of the expected changes after the KPIs training is that employees discuss metrics, not just use the list of indicators suggested by the management. After a certain period, we could interview team leads to see if that behavior pattern was successfully implemented.
When to measure? 1-2 months after the training.
The role of the software:
- Visualize expected improvements (on the process maps)
- Track behavior indicators on the scorecard
- Analysis function will help to track changes in the performance
Level 4. Impact
In the previous step, we could see the changes in the behavior of employees.
- For example, a workshop might be dedicated to the adoption of a new agile technique, and we could see that the employees actually started using those techniques (Level 3. Behavior).
Such shifts in the behavior are important and demonstrate that the trainer was able to influence trainees well enough to inspire them to change, but this doesn’t imply that the change was of any use for the organization.
The bottom-line of any training or a workshop is whether it helped the organization to execute its strategy more efficiently and effectively or not.
We could start searching for indications of such improvement several months after the training, but it is wiser to think about the expected impact and how we are going to measure it beforehand (in the KPI system, we call this approach “measurement by design”).
When to measure? 4-6 months after the training.
The role of the software:
- Present the company’s strategy map,
- Visualize strategic business goals, and
- Align indicators with business goals.
Popular Training KPIs don’t Help
While some KPIs are popular, it doesn’t mean that you want to have them on your dashboard.
What about training KPIs? The objective of any training program is to develop new skills and improve the efficiency of an employee. Let’s have a look at some popular KPIs and if/how they can help us in this concern.
|Why you DON’T need it
|Average training cost per full time employee, $
|This KPI shows how much we invest in the training of each employee, but it doesn’t tell us anything about the results that we expect. If training was successful? Has an employee learned something new that allowed them to improve efficiency?
|Average number of hours per full time employee, hours
|The same problem as with the previous KPI. Your employee might study full-time for the MBA program, but will it help your employee deliver better results, and to what extent?
|% of HR budget spent on training
|Using this KPI, we could have some approximate idea about a balance in HR department. If they spend 90% of the budget on finding new candidates, then probably your training program won’t be the best one.
Still, there is no required % of budget that the company must spend on training. Excellent training might cost just 1% of your budget but will result in high productivity growth.
|% of employees trained
|This KPI will tell you about the number of employees that were trained. It makes sense using this KPI when you know the effectiveness of certain training. But again, what was excellent for employees of one department might not work for employees of another.
|Employee training satisfaction index
|The problem of this KPI is that it is really subjective. I’m sure you know this feeling when you are inspired and excited by new ideas that you’ve just learned after another excellent keynote speech at some conference.
.In a few days, these ideas appear to be just engaging stories, which you cannot convert into the practice (learn how to change the situation).
|Training ROI, %
|To calculate a ROI, you need to know your investment (I’m sure you can calculate this) and financial estimation of benefits your business obtained as a result of the training.
That’s returning us to the initial question – how to calculate training efficiency. This KPI might work when objective training measures are available.
As you can see, the most popular training KPIs are a good fit for the corporate governance dashboard, but won’t really help to measure the effectiveness of the training session.
KPIs for eLearning
As part of crisis management response, many organizations decide to opt for distributed teams. When switching to the distributed team model, it is logical to choose the online format for the onboarding training for the new members of these teams.
What happens with the KPIs for the training in this case?
In their essence, the KPIs remain the same. What is going to change is the way the required knowledge and skills are delivered.
Let’s look at measuring the performance of eLearning from various perspectives.
At BSC Designer, we use the eLearning approach for our Strategy Execution Training and internally, for onboarding of the new hires to our remote team. Along with general ideas about measuring eLearning, I’ll share the best practices that we proved to work for us.
Instead of attending training at some physical location, the participants will use some online platform. In terms of financial impact, this means:
- Cost saving, $ (compared to the on-site learning)
- Online platform costs
These numbers alone won’t tell us anything new, but they will help to have a rough estimation of training ROI.
Engagement Metrics for eLearning
The online training should use different mechanics to engage with attendees.
The attention budget is no longer secured, as it happened with off-line events. The participants are no longer in the same room for a fixed time period.
The materials of the online training sessions might be pre-recorded, and the classical participation rate loses its relevance. With this idea in mind, we need to change the engagement metrics accordingly. For example, we can track:
- Watch time for e-course materials
- Access statistics for the training materials
Some eLearning products have a public landing page. This gives us the possibility of using social media metrics to understand if there is some emotional connection with the eLearning product. The basic metric to track in this case can be:
- The number of social shares (for a landing page of e-learning product)
Validating Business Impact of eLearning
What about validating the impact of eLearning? My recommendation is to make the process measurable by design.
Try to make important business outcomes to be the inevitable steps of the training.
How to do this? It depends on your eLearning product. For example, in the case of our online training, the participants don’t pass any formal test or exam. We organized online sessions around the idea of building a prototype of the Balanced Scorecard.
In our case, the KPI that validates the business impact is:
- The quality of the Balanced Scorecard prototype created, %
The created prototype is proof that the participant actually obtained the expected knowledge and skills. And we know this for sure because we see the results created by the trainee. Moreover, we can estimate the quality of the final prototype and suggest some improvement ideas.
Not all of the participants will get to the finish line. The reasons behind this fact require some attention from your side. In this context, a KPI to look at is:
- Abandonment rates, %
It is plural because we are interested in knowing what parts of the online training were the most challenging.
KPIs for Internal Changes
The eLearning format impacts the mechanics of the training sessions. An on-site training session is done in real-time, it requires a certain set of tools and materials. An eLearning format requires a different set of tools.
Besides having a private area with all training materials, we need to have convenient tools to track the progress of the participants, unlock the next steps of the training, conduct final exams, and finally generate the certificates.
The KPIs depend on the training process in your organization. Let’s formulate it in this way:
- Adaptation of the training mechanics to the online format, %
Finding Improvement Points for eLearning
One of the benefits of online training format is that we have much more analytics related to the participants of the training.
- We can detect the materials where trainees spend more time
- The most used search phrases for the training knowledge base
- View analytics by seconds for training videos
- Analyze the questions asked to the training facilitator
This gives us a leading indicator for the improvement of the e-learning:
- Analysis of training analytics, %
What If We Cannot Find Good Indicators?
That’s a typical situation with the performance indicators, and I’d like to highlight two points:
- Inability to find good performance indicators is often a sign that there is no clear understanding of how new behavior patterns are aligned with a company’s strategy.
- There is nothing bad about facing this problem. On the contrary, when facing this problem, you are one step closer to successful strategy execution.
The whole point of the KPIs exercise is to start asking the right questions, and you just got to that point. The first thing that I’d suggest is to revise the organization’s strategy map.
Understand your strategy better, and KPIs will come up naturally.
You will find more specific guidance in our KPI system.
Here is a summary of the discussed ideas:
- Measure training effectiveness on 4 levels – first impression, exam score, behavior changes, performance improvements.
- The cost of the measurement increases with the levels, as well as the benefit of measurement
- Understand expected performance improvement before the training
Finally, the main purpose of any measurement is not the measurement itself, but the behavior that this measurement induces.
By building a training scorecard, you invite all stakeholders into the discussion about the training program, its goals, its short-term, and long-term results. Such discussions help to focus training efforts on what actually matters for your organization.
Please share your feedback for this article in the comments, as well as your ideas about measuring training/workshop efficiency and effectiveness.