Evaluation - Gold or a Chimera?

In the film Top Gun, a rather fanciful account of the US Navy Fighter Weapons School, the Chief Instructor (callsign Viper) tells the hero who comes to him with a crisis of confidence:

 A good pilot is compelled to always evaluate what has happened so he can apply what he has learned. 

The resonance with organisational development is clear.

Although there are many models of evaluation the dominant one for learning and development interventions and equally applicable to coaching is Kirkpatrick’s.  He uses four levels of evaluation:

1.      reaction of client - what he or she thought and felt about the coaching

2.      learning - the resulting increase in knowledge or capability

3.      behaviour - extent of behaviour and capability improvement and implementation/application

4.      results - the effects on the business or environment resulting from the client’s performance

Sadly in too many instances Kirkpatrick is honoured in the breach rather than the observance.  Having been Head of Training in DFID for 8 years I can attest to this absolutely.  I designed evaluation protocols and models, carried out evaluations, and presented to the Cabinet Office on the politics of evaluation.  Not once were we able to get any further than level three and mostly never beyond level two.  In the main this was due to a number of factors:

• A non-trivial issue of attribution and causality, methodologically impossible to demonstrate without using control groups.
• A reluctance to use time series by the line: “That was six months ago – I am interested in other things now”.
• A concentration on the present and the future rather than the past in an organisation that had many policy changes and initiatives.
• A general scepticism as to whether evaluation could be in any way useful.

As a result we were never able to apply Viper’s Dictum in its entirety.  It was rather like “….evaluate what has happened so he [we]…can forget about it…”!  And the picture in the other organisations where I have been head of training was just as dismal for the same reasons. 

Coaching is no less worthy of evaluation than any other people strategy.  Indeed it can be argued that it is even more important to evaluate coaching because of its high relative cost and extreme individualisation.  Evaluation should be both normative, to provide information on techniques and coach effectiveness and summative to provide both the coach and the client (and his or her organisation) an estimate of the value of the process.

For coaches this means giving serious consideration to the appropriate evaluation mechanisms before coaching starts and even as far as being fairly specific during the tendering process.  To some extent the objectives of the coaching will be pre-agreed or in some intances established during the contracting process between the coach and the client and, as required, the sponsor.

For if we don’t evaluate we won’t know whether we have achieved what we set out to.