Authors
4 out of 5 companies do not get any significant impact from training employees or do not make any effort to measure and document whether the training has an impact. Yet, huge amounts of money are spent every year on sending employees on training courses where skills, tools and techniques are practised in a secure environment.
Why do so when no effect or impact is being traced? This article offers an example of how this can be done. Implement Consulting Group has been working in one of Denmark’s largest organisations from Q4 2010 to Q1 2012 with the purpose of designing training programmes by combining Brinkerhoff’s and Kirkpatrick’s thoughts on high-impact learning and measurement of impact on four different levels.
The purpose of this article is to contribute to an understanding of essential considerations about impact for an organisation in relation to starting up training programmes for employees. Furthermore, the purpose is to contribute to considerations about the organisational effort before and after the specific training courses.
Impact on different levels
To achieve business impact in change projects and transformations, a change in behaviour and an increase in capabilities will typically be required. A common way of increasing capabilities is to offer training to the persons concerned in order to increase their knowledge within the given area – in other words: tell them what they are supposed to do, why they are supposed to do it, and train them in how to do it. To measure the actual impact of the training, Donald L. Kirkpatrick describes four different levels of evaluation, each of which is a prerequisite for the next level:
- Reaction & planned action are the measured reaction/evaluation of how the delegates felt about the training or learning experience.
- Learning & confidence evaluation is the measurement of the increase in knowledge – before and after.
- Behavioural impact is the extent of applied learning back on the job – are the participants actually doing anything different after the training programme than before?
- Business impact is the effect on the business or environment by the participant as a result of the completed training programme.
This hypothesis is well-known and welldocumented. Evaluation models (many inspired by Donald L. Kirkpatrick) are widely used on levels 1 and 2 and focus on the individual perception and gain from the training, typically because these are the easiest measurements to perform using e.g. exit polls and tests. But how do we know if the learning actually leads to a behavioural change that in the end increases performance and benefits for the organisation?
Proving the hypothesis is very difficult using direct methods and first-order logic due to the vast number of variables. However, indirect methods (disproving the opposite, KPIs etc.) can build a very strong case – as long as they are established and agreed upon very early in the project, preferably before any training activities are executed.
Case study
The situation
The organisation in which the project was carried out is part of a shipping company and manages a budget of more than USD 400 million. The vast majority of the investments are allocated to project management and implementation of solutions in the business.
In order to be a highly successful project organisation, a standardised, broadly known and accepted project management methodology is required throughout the organisation.
A framework already existed and was applied by 18% of the project managers.
The complication
This lack of alignment had an impact on both portfolio and project management, which again had a negative impact on the Voice of Customer (VoC) KPI due to – among other things – miscommunication.
Ultimately, the bottom line was affected due to missed-opportunity cost. The lack of success of the existing framework was primarily (but not solely) due to four aspects:
- Size, complexity and lack of userfriendliness of the existing framework
- Absence of a common set of expectations to project manager behaviour
- No formal training in the framework was offered
- No organisational anchoring of the framework
The project
The objectives of the project were to provide a standardised, best practice approach to project management by providing:
- A common Project Management Model (PMM), language and framework
- A common understanding and utilisation of project management tools
- A common understanding of the role of the project manager and a set of common expectations between the various participants and stakeholders involved in a project
Before establishing deliverables, an impact case was built, addressing the desired impact on levels 3 and 4 (i.e. behavioural and business impact).
Level 3 impact goals:
- 80% of the functions utilise the PMM framework by the end of 2012
- 100% of the functions utilise the PMM framework by the end of 2013
- 95% of the project managers in the organisation utilise the PMM framework through the PMO site by the end of 2011
Level 4 impact goals:
- VoC KPI has increased by 2% by the end of 2012
- The project managers’ relative job satisfaction has increased by 4% by 2012
The impact case was qualified, challenged and adjusted over several workshops with significant stakeholders before being presented to the steering committee for the final decision.
The design
A very large part of the effort was focused on training the project managers in using the framework. In order to ensure impact on both level 1, 2 and 3, the below process was designed.
By the end of 2011, 106 participants had been trained.
Self-assessment
When signing up for the course, a confirmation letter with a self-assessment and an impact map was sent to the participant.
The purpose of filling in the self-assessment and the impact map prior to the training was to create focus and clear objectives for the learning they wished to achieve by participating in the training.
The impact map was a personal plan – created in cooperation with the manager – to ensure implementation and application of the PMM.
Pre-meeting
Based on the participants’ self-assessment of current competences and focus areas, a pre-meeting was held with the course participant, the manager and the course instructor.
The purpose of the meeting was to:
- Discuss the self-assessment
- Align expectations regarding the outcome of the course
- Align development goals with ongoing tasks for the project manager
- Support the manager in coaching towards continuous improvement of project management skills
Post-meeting
Two weeks after completion of the training and pre-/post-test, a follow-up meeting was held with the course participant, the manager and the course instructor. The purpose of this meeting was to:
- Discuss the outcome of the training based on the pre-/post-test
- Finalise the impact map
- Prepare a detailed personal development plan for the next three months
Impact meeting
After three months, an impact meeting was held with the manager, the course participant and the instructor. The purpose of the meeting was to follow up on the agreed goals and plans in the impact map and to establish whether the training had actually resulted in a change of behaviour.
The impact
Level 1:
Measuring reaction & planned action
Course evaluation was performed as a final session on the 2-day training programme. The participants filled in a course evaluation that measured:
-
Relevance of the course:
Indication of the extent to which it had been relevant to the participant and the extent to which the tools etc. could be applied directly after the training
-
Overall evaluation of the course:
Indication of the overall satisfaction with the course
Both parameters were evaluated on a 1-5 scale. On the continuously updated impact board, all data were gathered and summarised in the graph in order to monitor the progress.
Level 2:
Measuring learning & confidence
Two weeks after the training, the participants were asked to fill in a pre-/post-test. The purpose of this individual assessment was to identify the increase in knowledge as a result of the course. It consisted of 15 questions related to the five tools in the PMM framework. The scale was from 1 to 5, with 1 representing ”no knowledge” and 5 representing ”able to teach others” (see below graph).
Level 3:
Measuring behavioural impact
Short-term behavioural impact goals were set on an individual level. Immediately after completion of the pre-/post-test, the post-meeting was held with the participant and the manager, with the course instructor as the facilitator. Level 3 impact was not measured on a questionnaire or a 1-5 scale, but on the ability to follow the personal development plan, hence achieving the desired impact stated in the impact map. The results were that 100% of the participants who followed the training and implementation process (and at the same time had a current project!) followed their plans and implemented the framework.
So... was it worth it?
For each participant on the 2-day training course, three meetings were held: a premeeting, a post-meeting and an impact meeting – all involving the participant, the manager and the instructor. In addition, a substantial amount of work related to completion of various assessments, tests and impact map and the planning and organisation of the course was carried out.
Resources used in this approach are considerable, and it is time-consuming. It requires sustained commitment from both the sponsor, the project manager and, not least, the manager.
So what was the effect of the huge focus on impact? It is very hard to prove directly and document using first-order logic – as previously discussed. What we do know, though, is that all KPIs are pointing in the right direction:
- Level 1 evaluations on a 1-5 scale are
- Relevance of course >4.5
- Overall evaluation of course >4.6
- Level 2 evaluations on a 1-5 scale:
- Average score on the 15 questions increased from 2.9 to 4.02
- Level 3 evaluations document that goals formulated by the participant and the manager and documented in the impact map have been achieved
- Level 4 evaluations still await the yearly surveys, but there are clear indications that VoC as well as employee satisfaction have increased
Despite this, we cannot demonstrate an undisputable proof of business impact from the training, but we can disprove the opposite.
We do know at this point that without a significant effort before, during and after the training, no organisational effect would have been realised at all. At best, training alone has caused strong individuals to alter behaviour a little bit, but nothing substantial at an organisational level. Participants who only attended training and did not follow the entire process OR did not have a current ongoing project have shown little behavioural change.
On the other hand, focus on leveraged transfer, meaning to direct focus of the participant to the issues most likely to succeed and invite each participant to set goals that are current and relevant together with the participant’s manager, has resulted in all of the participants altering behaviour. Obviously some more than others, but all in the desired direction.
True impact is what would not have happened if we had done nothing. Some things we can measure and prove directly, some we have to believe in and at best prove indirectly.
