How to build a training scorecard that will help to separate time-spending on infotainment and highly useful performance-shifting training.
Training and workshops became a standard part of learning process in any organization. For my opinion the most important challenge for the organizations is to be able to detect the difference between less useful infotainment and get right to the point performance-shifting training. In this article we will discuss how to estimate the effectiveness of the training process and build a training scorecard.
Imagine a workshop that is delivered by a charismatic trainer, who knows the topic and easily engages with trainees. Sounds good, but we still need to ask a number of questions:
- Did this training give our team required knowledge, skills?
- Were those new skills actually applied?
- What parts of the training worked well, and what needs to be improved?
- How do we know that the investment in this training will finally lead to the performance improvements?
- Should we repeat this training? Should we hire the trainer once more?
These are the challenges that CEO/HR faces. Trainers in their turn, also need a data-powered feedback loop that will help them to improve their training products. They need to understand how to evolve the training program in order to increase the outcomes and the impact on the organization’s performance.
Kirkpatrick levels model
It is obvious that one-time measurement won’t provide us with the objective results. We will need to have several control points separated in time periods. The Kirkpatrick levels framework suggests to review a training from 4 perspectives:
- Reaction (after-training feedback)
- Learning (what was learned)
After some time we can track:
- Behavior (what was applied in practice)
- Impact (performance results achieved; effectiveness)
If the training is conducted by an 3rd party expert, then we also need to take into account a short-listing process. There should be some prerequisites for the trainers and training:
- How qualified the trainer is, and
- How the suggested training is aligned with company’s strategy.
The cost of the measurement
If we put these four steps on a chart with “Level” on one axis and “Difficulty to measure” on another, we’ll see that the easiest part to measure is the immediate reaction after the training, as measuring prerequisites requires some quantification of a screening process, but it is still doable.
For the rest of the levels the measurement efforts increases, as well as the cost of the measurement. This is one of the factors that justify the use of scorecards and automation tools.
Training scorecard template
Considering this article as an educational product, I’d like to share a template that one can use to evaluate training effectiveness in their organization. I’ll use BSC Designer software to demonstrate how this process can be automated.
First of all, I would suggest starting with a good strategy map that links your financial, customer-related, and internal goals with your learning initiatives. This will give a direction to all of the following management efforts.
The next step is to measure prerequisites. To do this we will need to quantify the following data:
- Trainer qualification
- Trainer communication skills
- Estimated training costs (travel expenses, accommodation, trainer fee)
- The match between training program and strategic goals
When quantifying these parameters make sure that the estimation is not done by a head of HR alone, ask key members of your team to share their opinions.
Measurement date: a few weeks before the training.
The software will help to:
- Assign weight to each criterion
- Create data series for several candidates
- Store individual evaluations, and the weights of the parameters
- Calculate the score for each of the candidates
Level 1. Reaction
At this level we need to get an after-training feedback that will help to get an idea about how engaging this training was. Some questions could give some data for the indicators:
- How do you feel about the training? (negative, neutral, positive).
- Would you recommend this training to your colleagues?
- Do you want to repeat this training in the future?
The list might go on, as more indications of high engagement can be found:
- The number of questions asked during and after the training
- Subjective estimation reaction of audience (were they participating, or checking emails on their smartphones?)
The software helps to:
- Collect the evaluation for each indicator shared by each participant
- Give natural language choice options
- Calculate the average scores
Measurement date: on the date of the training.
Level 2. Learning
The easiest way to quantify learning outcomes is to conduct a final test that will show how well the students understood the material.
The metrics could be:
- Exam pass rate, %
- Average exam score, %
- Before/after training test results
Again, the learning outcomes can be tested in different ways. For example, instead of using a simple test one can ask trainees to practice something that was learned. Respectively, the practice-related indicators will need to be implemented. In this way we are not only testing, but helping students to establish new behavior patterns.
Measurement date: at the end of the training.
We did the required minimum. The cost of the measurement for the next levels will increase. Should we go there? If we stop our measurement on this level then we are at risk of “false positive” situations, when a charismatic trainer delivers an engaging talk, that is well accepted by an audience (a kind of infotainment), but will rarely result in performance improvements in the organization.
Level 3. Behavior
Conducting tests is a good starting point to make sure that the trainees have obtained required knowledge, but what we actually want to see is a certain change in their behavior patterns. The quantification is this case depends on the nature of the process.
In some cases the indicators of new behavior will be as specific as:
- Time reduction achieved, %. If the goal of the training was to do something more effectively or simply faster, then you might track it by a KPI.
- Cost reduction achieved, %. If the goal of the training was to do something in a more effective way then you might track direct cost reduction, which is result of the training.
- Increased quality or more specific Reduce of the return problems rate, %. The KPI is applicable where the problem is clearly defined and linked to the final result. By the way, a repeated problem is one of the negative drivers of customer loyalty. If you will be able to decrease this number, you’ll make your customers happier.
- Performance improvement, %. The KPI must be linked to a specific performance index. For sales people it might be the number of sales generated, for a copywriter it might be readers to leads conversion.
In most cases we will need to spend time finding good indicators of new behavior patters. Our KPI system will help to frame the brainstorming process.
It is a good idea to ask trainer beforehand about expected changes in the behavior and the best way to make sure that they took place in the organization.
- For example, one of the expected changes after the KPI training is that employees start a discussion about metrics on the planning stage. After a certain period we could interview team leads to see if that behavior pattern was successfully implemented.
Measurement date: 1-2 months after the training.
The role of the software:
- Visualize expected improvements (on the process maps)
- Track behavior indicators on the scorecard
Level 4. Impact
On the previous step we could see the changes in the behavior of employees.
- For example, a workshop might be dedicated to the adoption of a new agile technique, and we could see that our employees actually started using those techniques (Level 3. Behavior).
Such shifts in the behavior are important, and demonstrate that the trainer was able to influence trainees well enough to inspire them to change, but this doesn’t imply that the change was of any use for the organization.
- The bottom-line of any training or a workshop is whether it helped the organization to execute its strategy more efficiently and effectively or not.
We could start searching for indications of such improvement several months after the training, but it is wiser to think about expected impact and how we are going to measure it before hand (what was called “measurement by design” in the KPI system).
Measurement date: 4-6 months after the training.
The role of the software:
- Present company’s strategy map,
- Visualize strategic business goals, and
- Align indicators with business goals.
Popular training KPIs don’t help
In the previous articles we discussed some popular HR KPIs like Turnover and Time to hire. Being popular doesn’t mean that you need to have them on your dashboard. What about training KPIs? The objective of any training program is to develop new skills and improve the efficiency of an employee. Let’s have a look at some popular KPIs and how they can a help us in this concern.
- Average training cost per full time employee, $. This KPI shows how much we invest in the training of each employee, but it doesn’t tell us anything about the results that we expect. If training was successful? Has an employee learned something new that allowed them to improve efficiency?
- Average number of hours per full time employee, hours. The same problem as with the previous KPI. Your employee might study full time for the MBA program, but will it help your employee deliver better results and to what extent?
- % of HR budget spent on training. Using this KPI we could have some approximate idea about a balance in HR department, if they spend 90% of the budget on finding new candidates, then probably your training program won’t be the best one. But still there is no required % of budget that company must spend on training. Excellent training might cost just 1% of your budget, but will result in high productivity growth.
- % of employees trained. This KPI will tell you about the number of employees that were trained. It makes sense using this KPI when you know the efficiency of certain training. But again, what worked excellent for employees of one department might not work for employees of another.
- Employee training satisfaction index. The problem of this KPI is that it is really subjective. I’m sure you’ve known this filling when after another excellent keynote speech on some conference you are inspired and excited by new ideas that you’ve just learned. In a few days, these ideas appear to be just engaging stories, which you cannot convert into the practice. This KPI might work, but it is better to measure the index 3 months after the training.
- Training ROI, %. To calculate a ROI you need to know your investment (I’m sure you can calculate this) and financial estimation of benefits your business obtained as a result of the training. That’s returning us to the initial question – how to calculate training efficiency. This KPI might work when objective training measures are available (see a paragraph below).
What if we cannot find good indicators?
- Inability to find good performance indicators is often a sign that there is no clear understanding of how new behavior patters are aligned with company’s strategy.
- There is nothing bad about facing this problem, on the contrary, when facing this problem, you are one step closer to a successful strategy execution.
The whole point of the scorecard/KPIs/maps exercise is to start asking right questions, and you just got to that point. The first thing that I’d suggest to do is to revise organization’s strategy map. Understand your strategy better and KPIs will come up naturally. You will find more specific guidance in our KPI system.
Here is a short summary of the discussed ideas:
- Measure training effectiveness on 4 levels – exam score, first impression, behavior changes, performance improvements.
- The cost of the measurement increases with the levels, as well as the benefit of measurement
- Understand expected performance improvement before the training
Finally, the main purpose of any measurement is not the measurement itself, but the behavior that this measurement induces. By building a training scorecard you invite all stakeholders into the discussion about the training program, its goals, its short-term, and long-term results. Such discussions help to focus training efforts on what actually matters for your organization.
Please, share in the comments your feedback for this article, as well as your ideas about measuring training/workshop efficiency and effectiveness.