I wrote some papers and article on measuring the effectiveness of training (including impact and ROI) back in 2005. Readers can see the link here: Return on Effectiveness (ROE): A Working Model for Measuring Effectiveness of Service Training in which I conceptualized a simple, intutive and practical 4-tier “Return on Effectiveness (ROE)” model for measuring training effectiveness and ROI as an engineer would do. Since then I have received quite many requests, particularly from technical training managers to elaborate the approach a bit more. Somehow, the approach I originally published showed quite an instant connection with a technical audience who had not a whole lot of business responsibilities directly. Sometimes Kirkpatrick’s Training evaluation model and Phillips’ ROI models, though widely used, either turn too abstract or turn too infeasible for engineering managers to even apply. This post is not an attempt to discredit existing models of training effectiveness or ROI measurement. The model may initially look like generic models like Kirkpatrick’s model. However, the attempt here is not to over-confuse new technical training managers with training lingo.
All in all, I believe professionals and training managers needed a ‘common-sense’ measurement approach – a simple feasible way to measure the effectiveness of the training. The model which I proposed was geared towards the language of ratio and indices, and a practical way to implement this approach which makes an instant connection with technical training managers. In this post, I will share 4-tier workable model of measuring training effectiveness of complex training operations with the focus on briefly how to implement or use it.
The whole picture
While the model below is described from a training standpoint but the moment we talk about on-the-job measurement, it is a collective measurement of learning and on-the-job performance which could be a result of several other support factors. In complex training dynamics and factors impacting the outcome may be hard to isolate as the sole effect of training per se. The basic premise of the approach described in this post is that training does not work in isolation. It has a close connection with other support functions like direct management of the employee, performance support systems and continuously learning that happens post-training. The business performance of any company comes as a result of learning and performance of employees, not just because of training. Therefore it should actually be a measurement of learning impact in the business rather than training alone.
The general outline of the model is described in the figure. It has 4-tiers, in line with most popular training effectiveness models. The model contains 4 tiers.
- Tier-1: Reaction index
- Tier 2: Improvement index
- Tier 3: Effectiveness index
- Tier 4: Impact index
4-Tier Training Effectiveness Measurement Modle (Image copyrights: Raman K. Attri)
Tier-1: REACTION INDEX: How closely training program meets the expectations of trainees?
- Objective: Measure training reaction of employees in terms of perceived skills/knowledge
- Who: Training Participants
- When: At the end of the training
- Outcome: Reaction Index (RI)
- Source: End-of-the training self-assessment and pre-training baseline
This tier measures the training reaction – a feedback score collected from each trainee at end of the class using a well-drafted survey form. You might say –what’s new. You have been doing all your life. No. what you might have been doing is measuring the reaction of trainees how they ‘feel’ about the training.
In complex business settings, your goal is to get how soon you can get your trainee ready for the job. You are interested in the results which give that impact on the business. You would not want to go smiley sheet ‘how well trainer taught the class and whether or not objectives were clearly stated or whether or not the material was comprehensive’.
The thing is that employees come to a training event to learn on-the-job skills that can make then successful in short time. That’s what the ‘expectations’ are. And what you want to do is to measure how well you meet the expectations. With changing business needs, what makes sense is to measure the instant confidence gained by trainees from the training even on the on-the-job skills required by them to do their job. At this stage, you can use participant’s post-training self-assessment of skills (knowledge and attitudes as well) they just acquired in the training in terms of their “perceived” confidence and competence level. These scores can be normalized or calibrated by instructor’s measurement of their skills as indicated by training assessments. This will provide a quantitative measure of their readiness for on-job assignments for which they came to training.
Put it simply, the higher the scores are, the closer you are to meeting the business needs. Remember that these employees are the one who help you meet the business needs in the field and your business needs cannot be separate from their on-the-job skill needs or their on-the-job skills needs cannot be different from your real business needs.
Applying the model
In order to use this information to improve your training alignment with business, you need to run the exact same set of the survey before trainees come to the training on exact same skills/measures. That data will give you the baseline scores of their pre-training skills and other performance measures. With the perceived confidence and competence scores and baseline pre-training scores available to you, you can compute an index called Reaction Index.
REACTION INDEX = ratio of post-training perceived skills score vs. pre-training baseline score
Making Training Decisions
- Ideally, Reaction Index should be > 1.
- A large Training Reaction Index will mean that training program has high relevance with the participants and they see the training aligned well the way they see their on-the-job challenges. Remember that RI is based on the perception of the participants and it may not hold reality. However, it is an important measure.
- What if Reaction Index < 1? It means that you need to re-assess the scope of the training and critically review your measurement instrument to see if it reflects on-the-job requirements correctly.
Tier-2: IMPROVEMENT INDEX: How well training improved on-the-job skills of trainees?
- Objective: Measure observed improvement in on-the-job skills as a result of training
- Who: Employee Supervisor
- When: At the end of 3 months post-training
- Outcome: Improvement Index (II)
- Source: Business unit employee performance data sources
Now since the employee has started doing his job and applied skills he learned during training, a supervisor is bound to see visible indicators of changes in job behavior, skills, and attitudes. Though there are several factors, including the influence of direct manager, responsible for this, the changes in skills and attitude in first 3 months can be attributed to training. The 3-months are just good enough period to assert the improvement due to training program but short enough to establish long-term effectiveness of the course or its impact on the business. Therefore you need an intermediate measure called ‘improvement’ with a measurement instrument (survey or through job-performance data) to be put in place 3-months post-training.
The complex organizational settings make use of single instrument infeasible. Therefore you may have to use an integrated system which combines data inputs (qualitative and quantitative) from various sources like company CRM, escalation matrix, task trackers, customer satisfaction scores, stakeholder 360 degrees, assignment data and so forth. The data from these sources can be used to get the supervisor to rate the skills (knowledge and attitude included if applicable) learned in training. If only skills are involved then the data can be consolidated to reflect supervisor’s rating on confidence and competence of the employee on all the skills received during training. The instrument can be subjective or objective or can be the one in which subjective measures are an objective measure.
The instrument is primarily for the supervisors and should be aligned very-well with the baseline instrument and the instrument used during the training program to measure expectations/ reaction of trainees. If you see, we have isolated it from any “business” measurement at this point. Let’s first establish training indeed resulted in improvement of skills and impacted the trained employee before we start measuring the impact on the business.
What you get out of this is the measure of observed performance of the employee in terms of skills (not in terms of business indicators) in the field. This data would show you improvement after training and how it is improving over the time. This exercise will give a “relative” improvement before and after training in terms of meaningful “acquired & applied” skills, improvement in the attitudes, enhancement of knowledge, level of handling of assignments and level of fulfillment of supervisor’s expectations.
Applying the model
You can use this information to compute the Training Improvement Index for the training program.
IMPROVEMENT INDEX = Ratio of post-training observed skills score vs. pre-training skills baseline score.
Making Training Decisions
- An ideal index would be > 1. Larger the ratio better the improvement training has created.
- What an Improvement Index of 1 means? It means training has not shown any improvement in the skills yet. This could be alarming – you need to watch it.
- You can compare this value to Reaction Index which was computed using the same set of skills. By rule, due to field exposure and increase in confidence, the training improvement index should be more than Training Reaction Index.
- What if Training Reaction Index < Improvement Index? You have an organizational trouble for which training is no longer a solution. Possibly the post-training performance support system and other supporting mechanism do not reinforce the learning in the training. This also tells another story that training may not have correctly designed in line with field requirements or field requirements may have changed since training.
- Combination of Reaction Index and Improvement Index will give you assessment of your strategy and if you need to revisit your training program in light of new information.
Tier-3: EFFECTIVENESS INDEX: How effective is the training in changing employee performance parameters?
- Objective: Measure impact of training on employee performance indicators
- Who: Employee Supervisor
- When: Quarterly
- Outcome: Effectiveness Index (EI)
- Source: Business Unit employee performance data sources
By now you already have a good idea about improvement in skills training kicked in the employee group who received the training. Assuming that ratio is more than 1, you need to move forward to see if employees have produced meaningful and measurable target results using his skills learned during the training program. This is supervisor and BU owner’s assessments on the quantifiable impact on employee performance indicators due to training.
At this stage, instead of numeric scale number based on subjective judgment, a data-driven approach is taken. Data can be collected from organization or BU’s CRM/ SAP or other integrated database repositories which tracks performance data for each business transaction and which can be broken down to an employee. Some of the examples of parameters indicating an employee’s performance are Service time, time to repair, time to close escalations, Productivity increase, Number of escalations handled, Case closure time, the success rate in closing support calls, etc. This data should be gathered as a baseline as pre-training exercise and later should be gathered on a quarterly basis.
The employee performance parameters used here should be the one which was used to analyze and draft the learning outcomes of the training program which generated the skill inventory used for pre-training baseline measurements. The tough part is mapping the skills to the on-the-job performance indicators. However, if a good job has been done to ensure training objectives reflects real-world performance of the employee group, then the mapping may be easy.
The goal of this tier is to gather nearly “quantified” impact of the training on key performance factors like time to service, man-hours, service cost involved etc. Note that at this point we are still not getting into business unit’s business performance collectively. If you want to measure the effectiveness of the training you need to first see the impact of training on the employee performance indicators. In short, it is called effectiveness of the training in terms of making employees more productive for business needs.
Applying the model
Above information can be used to compute the Effectiveness Index if the pre-training employee performance baseline data is available. This index measures how much training contributed or impacted to meet the individual employee’s performance goals as business contributor
EFFECTIVENESS INDEX = Ratio of post-training numbers vs. pre-training average employee performance baseline data
What if an employee is new and baseline data does not exist? Well, in that case, organization’s average baseline data can be used. In some instances, it is advisable to sue organizational baseline data on average to take an integrated and business approach.
Making Training Decisions
- Ideally, you want Effectiveness Index > 1
- A large value of this index indicates good translation of training into measurable employee job performance indicators. It also indicates that training program is designed correctly and is effective.
- What if Improvement Index is high but Effectiveness Index is low? In that case, you need to analyze the mapping between training objectives and on-the-job performance indicators.
- What if Improvement Index is low but Effectiveness Index is high? This is likely not possible. You would have gone back to my recommendation of Improvement Index being low and fixed the training before any other action. If the situation indeed happens, then training is not the contributory factor in improving the employee performance parameters.
Tier-4: IMPACT TRENDS: To what extent training impacted the business metrics specific to an employee group, BU or organization as a whole?
- Objective: Measure impact of training on employee group or BU or organization business KPI
- Who: BU owner
- When: 6-monthly or quarterly (whoever lines up with business measurements)
- Outcome: Impact Trends
- Source: Business unit and organizational performance data sources
If above 3 tiers are in place, then BU will have enough data which can be averaged and normalized quarterly over the entire organization for various training programs and for various business units. Now the three indexes computed (averaged or normalized) above can be plotted quarterly in relationship to the key performance measurements or metrics of the given business.
Every organization has some sort of metrics in place for tracking their performance. Some examples of business KPI are: Start to close Service time per call, Cost per call, Man-hours per case, Number of cases handled per engineer, Success rates of cases, Cost of labor per call, etc. During training analysis exercise, the training objectives are mapped to the business indicators. Not all business indicators are used to develop the training program and hence not all parameters are required to compute the impact of the training program. For example, for a repair related business, the measurements should be related to a number of cases handled, revenue from service, average hours spent in service, contract margins and so forth as opposed to the organizational P&L. If the training program was aligned with BU’s objectives then it makes sense to use BU business performance data. Depending on business unit’s business model, it is possible to compute Impact Index in relationship to some critical measurements. For example, if in a business service call duration (SCD) is a critical business-driving factor, then the ratio of this SCD can be plotted with respect to two parameters: BU’s baseline data on SCD vs. quarterly average on SCD and Quarterly MTTR data vs. Effectiveness Index. In a nutshell, the possibilities are endless depending on the nature of the business. Measurement cycle should line up with the measurement cycle for the business. If a business is monitored quarterly then plotting this relationship makes sense quarterly.
Summary of Steps in Data Collection
Above approach involves a good amount of systematic data collection. This requires implementation of a framework for such data collection to as much objectivity as possible.
Step 1: Pre-Training baseline data
– Employee performance indicator data
– Employee self-assessment on skills targeted in the training program
– BU or organizational baseline data on key business performance indicators (if available/ applicable)
Step 2: End-of-the-training program self-assessment data of each participant
Step 3: 3 months post-training supervisor’s rating on participant’s on-the-job skills
Step 4: Quarterly employee on-the-job performance indicator data
Step 5: Quarterly or six-monthly BU or Organizational key business performance indicators/matrix
For a detailed procedure on data collection, refer to the Return on Effectiveness (ROE): A Working Model for Measuring Effectiveness of Service Training.
As a note, this model was developed back in 2005 and has not been updated with more contemporary challenges of today’s business. Readers are advised to modify or adapt the model to their specific requirements. In general, I have applied the model in the context of complex technical skills and it works without much overhead other than instituting systematic data collection. In fact, the model expands thinking to stay aware of data that may be needed to implement such a model. Further, since model presents the results in form of indexes, which may be easily trackable and easy to explain to upper management, rather than chasing the ROI in terms of investment and outcome with actual dollars.
What do you think of this approach? Do share your thoughts in the comments box.
- Attri, Raman K. 2005, “Return on Effectiveness (ROE): A Working Model for Measuring Effectiveness of Service Training”. https://dx.doi.org/10.13140/RG.2.1.1085.5765 and https://www.researchgate.net/publication/285587862.
- Kirkpatrick, DL & Kirkpatrick, JD 2009, Transferring Learning to Behavior: Using the Four Levels to Improve Performance, Berrett-Koehler.
- Kirkpatrick Partners, (no date). The Kirkpatrick Model. https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
- Vance, A. 2017. The Bottomline on ROI: How to Measure the Results of Your Training, ROI Institute. https://roiinstitute.net/bottomline-roi-measure-results-training/
Image credits: Pixabay CC0 attribution
Image of “ROE” model- copyright Raman K Attri
Article originally published: Sept 11, 2014