Training Measurement Part 3 – Now What?

So training measurement isn’t perfect. What to do?

• Continue to measure but understand what you’re actually measuring or not measuring. Being realistic with the use of measurement results provides more limited but more useful data. For example, Level 1 can show whether an instructor was liked enough to engage the group. If the instructor isn’t liked, it’s more likely that the training won’t be retained and used. Level 1 can also weed out “training bombs” because generally the same group of employees is evaluating all of your training efforts. That is, over time a reasonably consistent group of employees generates a “baseline” evaluation. If there’s a big negative deviation, something is wrong.
• Accept that training is likely a component and not the sole cause of business improvements. Have faith that training is directionally the correct thing to do, even if the exact results are somewhat obscure.
• Worry less about measurements and more about what it takes to meet the specific needs of the business. Relevant training that helps people do their jobs better will be effective. Irrelevant training that wastes their time will not be effective. Make sure the training you do is “right.”
• If you accept that people want to do a good job, then you also should agree that they will apply relevant new skills to the job. So don’t just do the right training, but remove all the obstacles that prevent transfer of new skills to the job. This means providing a blend of learning opportunities that meet various needs and provide skills using the technologies that people are become accustomed to.

Training Measurement Part 2 – The Reality

Last time we talked about the typical measures of training effectiveness. How do they work in the real world?

Level 1 (reaction): Post-training evaluations usually ask participants to rate the training material and instructor for relevance, completeness, clarity, etc. These evaluations could give good information if diligently completed by each participant. More often the responses reflect overall “feelings” about the experience that include the instructor’s style and the quality of the lunch. As a result, these evaluations are good for comparing one course or instructor with another with the same participant group. If the question “how likely are you to use this training” is asked, almost all participants say “very likely.” But remember, retention is only 2-20% so intentions are unlikely to turn into on-job behaviors.

Level 2 (learning): Pre-training and post-training tests work very well when the learning objective is the knowledge of facts. For example, the knowledge gained about the history and heritage of a company can be assessed this way. There’s little or no correlation of such test results to use of the knowledge on the job. It might be used; it might not be used.
Levels 1 and 2 measure aspects of the training’s effectiveness, but are not able to directly assess training’s results.

Level 3 (observation): At this level results-measurement begins to work, but the work also begins. One approach is to have an observer actually watch to see if a new behavior is being used. Three problems occur here. First, observed employees naturally behave differently if they know they’re being watched. Second, observation doesn’t work well for “mental behaviors” (how do you watch a person think and decide if the thinking behavior is different?). Third, successful observation today doesn’t mean sustained behavioral change tomorrow. Sometimes third party “observation” works well. For example, ask customers or colleagues to assess behaviors over time and look for statistically relevant improvements. Auto dealers routinely call after a vehicle was serviced to ask questions about the customer’s treatment. Over time multiple answers can show trends in behaviors.

Level 4 (results): This is the level that really counts. If we can see a bottom line result that comes from training, great. Remember that training isn’t the point. Getting results is the point. Businesses have many financial and other (e.g. inventory age) performance indicators. Unfortunately, they may not tell you much about training. Here’s why. Training results often take time to manifest themselves in business results. In the meantime, many variables in the business may have changed. Is the profitability improvement in that division result from training, or changed market conditions, or new personnel, or changing consumer tastes, or the failure of a competitor? Most likely, the results are a combination of many such factors.

Next, what to do?

Training Measurement Part 1 – The Objective

Training isn’t the objective. It’s just a tool. Improved business results is the objective. If we train people in the wrong things using the wrong methods, we might feel good that we’ve completed our training goals, but nothing will change.

In an earlier segment we shared that ASTD participants in San Diego this spring thought that 2-20% of training was retained. I’d really want to change what I was doing if I was getting 5 tablespoons of gasoline when I paid for a full gallon (2% for my money). At the very least I’d want 20%, not 2%. So I’d measure how much gas I was getting. Likewise, I’d try to measure my training results.

The generally accepted method of training measurement uses four levels. It’s pretty solid theoretically, but our experience – shared by most people we’ve talked to – is that reality is less satisfactory. The four levels are:

• Level 1 – Participants react to the training through questionnaires, comments, focus group sessions and the like. Typically some form of scale is used (really like, like, dislike, etc.) to enable comparative measurements.
  
• Level 2 – Participants’ degree of learning is measured by comparing pre-training and post-training test scores. On-job assessments and supervisor reports can also be used.
  
• Level 3 – Desired behavior change in participants is measured through self-assessments, on-job observations and customer reports. This level is difficult to quantify, but attempts to look for useful results.
  
• Level 4 – Improvements in business results are measured, usually by using financial or inspection data.

In our next entry, we’ll look at how this usually works in the real world.

Blended Learning – Part 2: The Why

Why are companies “blending” their learning strategies? There are at least four reasons:

• Skills are available over a longer time period. For example, classroom training is a one-time event. But when augmented by performance support tools, for example, the skills can be available forever.
  
• A variety of training methods meets more of your employees’ needs. Some prefer the give-and-take of a classroom. Others want answers right now, or don’t have time for a lot of formal training.
  
• Blending permits companies to devote each training method to the things it does best, instead of expecting it to do all things well. Classroom training facilitates the discussion of personal experiences – useful for changing fundamental attitudes about a subject, for example. Performance support provides immediate answers without trying to be a forum for discussion and contemplation.
• Blending allows companies to target training methods to its demographics. Younger employees may prefer one style of training, and older employees may prefer another.

Blended Learning – Part 1: The What

More and more companies are moving towards “blended learning” strategies. Blended learning has several definitions, but in general means that more than one type of learning method is used. Blended learning combines traditional classroom and e-learning training methods with group activities, self-guided project activity, reference materials and performance support.

Blended learning approaches can be organized into three categories. The most conventional is instructor- or expert-led training. This method works well for “hard skills” (for example, learning a specific set of process steps necessary to operate a machine - the instructor is right there to correct errors and answer questions). It works well for transferring knowledge that would be difficult for an individual to learn (e.g. calculus). And it’s great when a group of participants can benefit from sharing experiences and opinions. On the other hand, it’s expensive, time consuming, and the jury is still out on whether “soft skills” are transferring to the job. (See an earlier edition of this Blog).

The second category of blended learning methods – group-oriented activities – is less common. In these methods the group is self-directed without a lot of input from “experts.” Quality teams used this method of learning, although they were not traditionally thought of as “learning activities.” Today, companies often use “development jobs” or “project teams” where learning can occur as work gets done. Role play can also fit this category.

The third category is individual-led methods. E-learning has been the mainstay of this category. However, performance support is becoming much more important, as are coaching-mentoring programs and help-desks. Next time…why Blended Learning?