Myths about Evidence Based Training

There’s a lot going on in the world of pilot training right now. In particular, there is some excitement about the introduction of ‘Evidence Based Training’ (EBT). EBT is a new approach to pilot training that has the potential to provide significant safety benefits. There’s also a lot of misunderstanding and misinformation. Here are some ‘myths’ that I’ve heard:

“We’ve bought in an EBT solution”

One important element of a project to implement EBT will be to select or develop software that can support data collection and analysis. Buying in the software doesn’t provide you with EBT any more than buying a word processor will make you a best-selling author.

“As long as we check pilot’s competencies we don’t need to train every manoeuvre”

Seriously? As a pilot, I may have excellent stick-and-rudder skills (flight path management manual, FPM) but if I’ve never practised how to do a two-engine go-around on an aeroplane with under-slung engines and conventional controls then there’s a good chance that it’ll go wrong. If it does go wrong then giving me additional training in other elements of FPM (say medium turns and landing techniques) isn’t going to help.

Competencies are a good way of describing what pilots do, but that doesn’t change the fundamentals of how to train someone to fly an aeroplane.

“We don’t need to use ‘absolute’ standards in grading”.

When instructors make assessments they are likely to grade relative to all the other candidates they assess. This produces a norm-referenced assessment where individuals are graded relative to the wider group. This has some value but it does not allow any evaluation of how effective the training programme is overall. If the standard of all candidates changes then the overall grade won’t change. One of the objectives of grading in EBT is to provide data to a ‘training measurement system’ that can be used to update and improve the programme. This means that assessments must be based on some external criteria.

“We’ve got great inter-rater reliability”

Having good inter-rater reliability (or concordance) is vital to an EBT programme. The pilots need to know that there will be a high level of consistency in competency assessments made by different instructors. Nothing undermines confidence like being assessed as ‘good’ one day and then doing the same thing another day and being assessed as ‘unsatisfactory’ (it happens).

An operator cannot assess inter-rater reliability until the operator has some means to measure it. Most operators don’t. I hear things like ‘all our instructors grade within plus or minus one grade’. For a pilot ‘plus or minus one grade’ is the difference between business a usual and looking for a new career while wondering how to feed the kids. We need to do better than this.

There are statistical techniques that can be used to make a proper assessment of inter-rater reliability.

“We can provide an approved EBT instructor course”

An important part of any EBT implementation project will be to ensure that instructors understand the EBT concept and, in particular, that they are proficient in making assessments using the operator’s competency framework. The training programme for instructors will need to be approved by the Competent Authority (as do all pilot-training programmes). The instructor training programme can’t be developed until the competency framework is finalised and the training needs of the instructor population have been established. Operators may choose to work with external experts to develop this training, but there isn’t a ‘one size fits all’ that can be pre-approved.

“Our airline uses EBT for type-rating training”

Type rating training requires pilots to be trained and to demonstrate competency in a set of manoeuvres. In European rules, the list of manoeuvres appears in Appendix 9 to Part FCL. This list of manoeuvres can be replaced by other items specified by the aircraft manufacturer in the approved ‘Operational Suitability Data’ (OSD). EBT is “Training and assessment that is characterized by developing and assessing the overall capability of a trainee across a range of Pilot Core Competencies rather than by measuring the performance of individual events or manoeuvres” [ICAO document 9995].

Some operators may be implementing some ‘EBT principles’ into their type rating courses (for example the use of a competency framework for debriefing) but it is not possible to have an ‘EBT type rating’ under current regulations.

“Business Jet Operators are implementing EBT”

Lesson plans for EBT recurrent training are developed according to the list of training topics in the Appendices to ICAO document 9995. The appendices contain different requirements for different generations of turbo-prop and jet airliners. The content of the appendices is based on extensive research and analysis (see the IATA Data Report for Evidence based Training). There is no appendix for business jets because the research hasn’t been done to establish which topics should be trained for business jets. Until this research is done EBT is not applicable to business jet operations.

Categories: EBT