Training Measurement Part 2 – The Reality

Last time we talked about the typical measures of training effectiveness. How do they work in the real world?

Level 1 (reaction): Post-training evaluations usually ask participants to rate the training material and instructor for relevance, completeness, clarity, etc. These evaluations could give good information if diligently completed by each participant. More often the responses reflect overall “feelings” about the experience that include the instructor’s style and the quality of the lunch. As a result, these evaluations are good for comparing one course or instructor with another with the same participant group. If the question “how likely are you to use this training” is asked, almost all participants say “very likely.” But remember, retention is only 2-20% so intentions are unlikely to turn into on-job behaviors.

Level 2 (learning): Pre-training and post-training tests work very well when the learning objective is the knowledge of facts. For example, the knowledge gained about the history and heritage of a company can be assessed this way. There’s little or no correlation of such test results to use of the knowledge on the job. It might be used; it might not be used.
Levels 1 and 2 measure aspects of the training’s effectiveness, but are not able to directly assess training’s results.

Level 3 (observation): At this level results-measurement begins to work, but the work also begins. One approach is to have an observer actually watch to see if a new behavior is being used. Three problems occur here. First, observed employees naturally behave differently if they know they’re being watched. Second, observation doesn’t work well for “mental behaviors” (how do you watch a person think and decide if the thinking behavior is different?). Third, successful observation today doesn’t mean sustained behavioral change tomorrow. Sometimes third party “observation” works well. For example, ask customers or colleagues to assess behaviors over time and look for statistically relevant improvements. Auto dealers routinely call after a vehicle was serviced to ask questions about the customer’s treatment. Over time multiple answers can show trends in behaviors.

Level 4 (results): This is the level that really counts. If we can see a bottom line result that comes from training, great. Remember that training isn’t the point. Getting results is the point. Businesses have many financial and other (e.g. inventory age) performance indicators. Unfortunately, they may not tell you much about training. Here’s why. Training results often take time to manifest themselves in business results. In the meantime, many variables in the business may have changed. Is the profitability improvement in that division result from training, or changed market conditions, or new personnel, or changing consumer tastes, or the failure of a competitor? Most likely, the results are a combination of many such factors.

Next, what to do?