Measuring the impact of training
4 out of 5 companies do not get any significant impact from training employees or do not make any effort to measure and document whether the training has an impact. Yet, huge amounts of money are spent every year on sending employees on training courses where skills, tools and techniques are practised in a secure environment.
Why do so when no effect or impact is being traced? This article offers an example of how this can be done. Implement Consulting Group has been working in one of Denmark’s largest organisations from Q4 2010 to Q1 2012 with the purpose of designing training programmes by combining Brinkerhoff’s and Kirkpatrick’s thoughts on high-impact learning and measurement of impact on four different levels.
The purpose of this article is to contribute to an understanding of essential considerations about impact for an organisation in relation to starting up training programmes for employees. Furthermore, the purpose is to contribute to considerations about the organisational effort before and after the specific training courses.
To achieve business impact in change projects and transformations, a change in behaviour and an increase in capabilities will typically be required. A common way of increasing capabilities is to offer training to the persons concerned in order to increase their knowledge within the given area – in other words: tell them what they are supposed to do, why they are supposed to do it, and train them in how to do it. To measure the actual impact of the training, Donald L. Kirkpatrick describes four different levels of evaluation, each of which is a prerequisite for the next level:
This hypothesis is well-known and welldocumented. Evaluation models (many inspired by Donald L. Kirkpatrick) are widely used on levels 1 and 2 and focus on the individual perception and gain from the training, typically because these are the easiest measurements to perform using e.g. exit polls and tests. But how do we know if the learning actually leads to a behavioural change that in the end increases performance and benefits for the organisation?
Proving the hypothesis is very difficult using direct methods and first-order logic due to the vast number of variables. However, indirect methods (disproving the opposite, KPIs etc.) can build a very strong case – as long as they are established and agreed upon very early in the project, preferably before any training activities are executed.
The organisation in which the project was carried out is part of a shipping company and manages a budget of more than USD 400 million. The vast majority of the investments are allocated to project management and implementation of solutions in the business.
In order to be a highly successful project organisation, a standardised, broadly known and accepted project management methodology is required throughout the organisation.
A framework already existed and was applied by 18% of the project managers.
This lack of alignment had an impact on both portfolio and project management, which again had a negative impact on the Voice of Customer (VoC) KPI due to – among other things – miscommunication.
Ultimately, the bottom line was affected due to missed-opportunity cost. The lack of success of the existing framework was primarily (but not solely) due to four aspects:
The objectives of the project were to provide a standardised, best practice approach to project management by providing:
Before establishing deliverables, an impact case was built, addressing the desired impact on levels 3 and 4 (i.e. behavioural and business impact).
Level 3 impact goals:
Level 4 impact goals:
The impact case was qualified, challenged and adjusted over several workshops with significant stakeholders before being presented to the steering committee for the final decision.
A very large part of the effort was focused on training the project managers in using the framework. In order to ensure impact on both level 1, 2 and 3, the below process was designed.
By the end of 2011, 106 participants had been trained.
When signing up for the course, a confirmation letter with a self-assessment and an impact map was sent to the participant.
The purpose of filling in the self-assessment and the impact map prior to the training was to create focus and clear objectives for the learning they wished to achieve by participating in the training.
The impact map was a personal plan – created in cooperation with the manager – to ensure implementation and application of the PMM.
Based on the participants’ self-assessment of current competences and focus areas, a pre-meeting was held with the course participant, the manager and the course instructor.
The purpose of the meeting was to:
Two weeks after completion of the training and pre-/post-test, a follow-up meeting was held with the course participant, the manager and the course instructor. The purpose of this meeting was to:
After three months, an impact meeting was held with the manager, the course participant and the instructor. The purpose of the meeting was to follow up on the agreed goals and plans in the impact map and to establish whether the training had actually resulted in a change of behaviour.
Course evaluation was performed as a final session on the 2-day training programme. The participants filled in a course evaluation that measured:
Indication of the extent to which it had been relevant to the participant and the extent to which the tools etc. could be applied directly after the training
Indication of the overall satisfaction with the course
Both parameters were evaluated on a 1-5 scale. On the continuously updated impact board, all data were gathered and summarised in the graph in order to monitor the progress.
Two weeks after the training, the participants were asked to fill in a pre-/post-test. The purpose of this individual assessment was to identify the increase in knowledge as a result of the course. It consisted of 15 questions related to the five tools in the PMM framework. The scale was from 1 to 5, with 1 representing ”no knowledge” and 5 representing ”able to teach others” (see below graph).
Short-term behavioural impact goals were set on an individual level. Immediately after completion of the pre-/post-test, the post-meeting was held with the participant and the manager, with the course instructor as the facilitator. Level 3 impact was not measured on a questionnaire or a 1-5 scale, but on the ability to follow the personal development plan, hence achieving the desired impact stated in the impact map. The results were that 100% of the participants who followed the training and implementation process (and at the same time had a current project!) followed their plans and implemented the framework.
For each participant on the 2-day training course, three meetings were held: a premeeting, a post-meeting and an impact meeting – all involving the participant, the manager and the instructor. In addition, a substantial amount of work related to completion of various assessments, tests and impact map and the planning and organisation of the course was carried out.
Resources used in this approach are considerable, and it is time-consuming. It requires sustained commitment from both the sponsor, the project manager and, not least, the manager.
So what was the effect of the huge focus on impact? It is very hard to prove directly and document using first-order logic – as previously discussed. What we do know, though, is that all KPIs are pointing in the right direction:
Despite this, we cannot demonstrate an undisputable proof of business impact from the training, but we can disprove the opposite.
We do know at this point that without a significant effort before, during and after the training, no organisational effect would have been realised at all. At best, training alone has caused strong individuals to alter behaviour a little bit, but nothing substantial at an organisational level. Participants who only attended training and did not follow the entire process OR did not have a current ongoing project have shown little behavioural change.
On the other hand, focus on leveraged transfer, meaning to direct focus of the participant to the issues most likely to succeed and invite each participant to set goals that are current and relevant together with the participant’s manager, has resulted in all of the participants altering behaviour. Obviously some more than others, but all in the desired direction.
True impact is what would not have happened if we had done nothing. Some things we can measure and prove directly, some we have to believe in and at best prove indirectly.
Seven things you should know about differentiation
Implement Consulting Group