Job trainers have a lot of things to check off their to-do list.
One is to evaluate the learning of employees who have completed training. This means things like assessments and tests, seeing if people can pass tests, have necessary knowledge, and (most importantly) have acquired necessarily skills/can perform necessary tasks.
The second is to determine if the training is having a positive effect on the relevant performance metrics for the company and, if possible, to determine an ROI for the training (this is how you’re going to really prove your worth and really prove your training is effective). In terms of Kirkpatrick’s training evaluations, we’re talking about the elusive but equally important Level 4 here.
But a lot of trainers go to school and learn a lot about instructional design theory while learning next to nothing about performance metrics (this includes me-guilty). As a result, it’s not always clear how to start showing if training has had a positive effect on those performance metrics.
To help solve this problem, we thought we’d give you an introduction to some of the theory behind the development of meaningful workplace performance metrics, and in particular to what are known as “balanced performance metrics.”
This will be one of a series of articles we’ll write on how training is related to performance metrics and KPIs not just for the training department but for the company as a whole, so keep your eyes on future articles for more on this topic.
What you’ll learn here is based off a handy little guide called Designing Metrics: Crafting Balanced Measures for Managing Performance by Dr. Bob Frost. We found this book to be really helpful, pleasantly brief, and to-the-point. We recommend you buy a copy if this article sparks your interest, and we note that Dr. Frost has written a few other books that look interesting as well. In particular, Measuring Performance: Using the New Metrics to Deploy Strategy and Improve Performance looks like it might be good and a logical next step to this book.
Frost begins his book by breaking performance metrics into two very broad categories:
The primary difference, according to Frost, are explained below.
Older performance metrics:
On the other hand, newer methods create performance metrics that:
Frost explains that these newer metrics are “balanced,” meaning that they look at many parts of a company’s performance instead of just focusing on financials. This allows companies to evaluate their performance from more perspectives or “angles,” to gain more insight into what they’re doing, and to better anticipate what they should be doing next.
We’ll learn more about these various balanced performance metrics in the next section.
According to Frost, measuring performance begins by selecting a measurement framework.
Your measurement framework(s) will help you identify your company’s critical success factors (CSFs), and your CSFs will help you identify what you measure (your key performance indicators, or KPIs).
Frost then presents six different balanced measurement frameworks. They are:
Let’s look at each in more detail, leaning on Frost’s explanations in his book.
The Balanced Scorecard is a popular framework for performance metrics. According to Frost, it’s the most commonly used at Fortune 500 companies, and Mobil Oil’s use of it is especially well known (it was critical for helping a once-floundering Mobil turn things around, according to Mobil execs, before they merged with Exxon).
Companies that use a balanced scorecard measurement framework should ultimately create performance measures in four different categories:
Let’s look at each in more detail.
Nuts and bolts financial measures.
How are your internal processes progressing?
This includes things like quantity, quality, speed, cost, and so on.
What do your customers think of you?
Customer satisfaction measurements, things like customer service wait time, etc.
These items ultimately address whether or not you can sustain growth and/or retain your current market position (or improve it).
Can include “nuts and bolts” things like training time, training completed, and training costs, but also things like major changes implemented and new strategic dimensions.
Want to know more about the balanced scorecard? Check out The Balanced Scorecard: Measures that Drive Performance by Robert S. Kaplan and David P. Norton, available at the Harvard Book Review.
Strategy maps are really just a “repackaging” of the balanced scorecard to make it clear how the balanced scorecard and the performance measurements lead to the business achieving its business strategy. The strategy map was developed by Kaplan and Norton, big players in performance measurement, in their book The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment.
A strategy map looks at the same four aspects that the balanced scorecard does, but with each leading to the next.This makes it easy to see the company’s strategy as laid out in the measurement framework.
From “bottom” to “top,” these are:
Let’s look at how each leads to the next.
This framework is a three-step process:
This one is more common in government agencies and non-profits and not used much at all in business.
There are different ways to use the program logic model. In one of the most common, you break things down into four categories:
Let’s look at each.
Inputs include resources use to “do” things (human, financial, facilities and technologies).
Activities include the things that get done.
Outputs are short-term results of getting those things done.
Outcomes are the ultimate, long-range goals of the organization or program.
The enterprise performance framework is fundamentally based on the recognition of economic pressures and competition within a market.
It looks at three aspects of a company:
Let’s look at each a little more.
Effectiveness: How well is the company (or organization) doing in terms of completing its ultimate goal or mission?
Efficiency: How efficiently is the company doing in terms of reaching its ultimate goal or mission? Or few (or many) resources is it using to make its accomplishments?
Strategic Improvement: What is the company doing internally to anticipate/adapt/change/prepare for the future and changing business realities?
As you see, effectiveness and efficiency look at the present (and the past), while strategic improvement looks at the future (and the present).
The cascading framework starts by identifying the company’s key mission or goals, then analyzes how the business units (and increasingly smaller sub-units) contribute to helping the company achieve that mission.
So it might look something like this:
The idea is that each business unit is evaluated based on how they contribute to the goal or the critical success factors (CSFs) of the unit above.
So, for example, level 2 may include 20 different mills at 20 different sites. Each mill would be evaluated based on how it’s helping the organization as a whole attain its goal(s).
Level 3 (at one of those 20 mills) might include departments such as Production, Quality, HR, etc. Each department would be evaluated based on how it’s helping the mill it’s part of achieve its critical success factors (CSFs).
And level 4 (at one of those 20 mills, in one of those departments, such as Production), might include multiple areas, such as Machine Line 1, Machine Line 2, and so on. Each area would be determined based on how it’s helping its department achieve its critical success factors.
The point is to make sure different business units don’t “go rogue/AWOL” and become so autonomous they’re not really contributing to the overall mission.
Now that you know about balanced performance measurement frameworks, let’s see how you can select one, then use that to develop a list of critical success factors (CSFs) and then use those to create your list of performance measurements/key performance indicators (KPIs).
So what’s the best measurement framework for your company?
There’s no one answer for that. Frost recommends getting to know them, analyzing which fits your circumstances best, and picking one, then moving forward to create your critical success factors (CSFs) and your actual measures/key performance indicators (KPIs). He stresses the importance of remembering that measurement frameworks true worth is in helping you select CSFs and KPIs.
Critical success factors are the things your company must do to have success and be successful in the future.
Once you’ve identified your performance framework, you can use them to begin identifying your CSFs.
For example, if you’re using the balanced scorecard measurement framework, and you’re considering the “Customer” category, CSFs include things like customer satisfaction.
Create a list of all the CSFs for your performance measurement and then move on to the next step.
Critical success factors are then used to determine what you should measure.
Again, let’s assume we’re using the balanced scorecard measurement framework. As part of the balanced scorecard, we’re identifying the critical success factors (CSFs) for our Customer level. We’ve determined that customer satisfaction is one of those CSFs.
We can then determine ways to measure customer satisfaction–such as “average time on hold or in line” and “percent of cases successfully closed.”
Again, there’s no single right answer here.
Frost suggests that you have enough measurements that you acknowledge complexity, but not so many that you get lost, and notes that not all measurements will be important at all times.
In addition, he warns you against the tendency to “rob Peter to pay Paul” when measurement systems are set up. If your company is going to measure something, know that there will be a tendency for people to try to meet that measurement even at the expense of other important things that aren’t measured. Keep them in mind and try to create a balanced measuring system.
We hope that was helpful. Let us know if you have any thoughts by writing in the comments section below. We’d be especially interested to hear what you learned about this in school (if you did) and what measurements your company uses.
Because this issue is so closely related to training evaluation for trainers, you may also be interested in some of the following articles:
Please feel free to download our FREE GUIDE TO WRITING LEARNING OBJECTIVES, below.
All the basics about writing learning objectives for training materials.