PPL e-Exam system in practice

Today I conducted my first theory e-Exam as examiner, after having witnessed a couple delivered by the CFI.

System Setup

UK PPL theory exams moved from a paper to computer based system in October 2020. Any registered ATO or DTO can apply to the CAA for authorisation to conduct them, but only authorised Ground Examiners can do so.

There are four separate websites involved in the administration and operation:

Student Admin	portal.caa.co.uk	Register with CAA Request membership of ATO/DTO View exam results
Examiner Admin	ato.caaexams.co.uk	Accept student memberships Book exams Pay for exams View results
Examiner Exam	quadrant-invigilation.caaexams.co.uk	Setup exam ready to start on another computer
Student Exam	quadrant-delivery.caaexams.co.uk	Used by student to undertake exams

Before any exam can be taken:

1. the student must register and be accepted on the CAA Portal (portal.caa.co.uk). This can take a few days
2. the student must request e-Exam services on the Portal. This typically happens within minutes
3. the student must request membership for e-Exams with their ATO/DTO
4. the ATO/DTO must accept membership

When the ATO/DTO confirms a student is ready to take an exam, the system requires two computers both running Chrome browsers and connected to the same local network – one for the examiner, the other for the student. I use my own laptop for the examiner station while the club provides a dedicated computer in the exam room for the student. It is also possible for student’s own laptop to be used and/or for multiple exams to be conducted simultaneously with different students. This generally runs fairly smoothly, although I have had some glitches where the office network configuration had been changed without my knowledge.

1. The examiner logs in to the examination admin, books and pays for the exam. The exam can be taken anytime in the following 90 days.
2. The examiner logs into the examination delivery website, selects the exam and computer to be used, and authorises it
3. The student will see the examination start screen on a separate computer and can start the exam when ready
4. The exam finishes either when it times out or the student chooses to end the exam early.

On completion, the system records the result and sends an email to the student, then takes a few minutes to process and update the results. The examiner can then login to the administration website, view the result including a knowledge deficiency report. This is a bullet list of codes relating to Learning Objectives which can be looked up on the theory syllabus. Examiners don’t see any of the specific questions but should debrief the candidate on any questions they were unsure of and especially those that might be in the deficiency report.

PPL Theory Study

In my experience, most PPL students self-study for these exams. In the past it was more common to attend weekly evening classes but few clubs now have the critical mass of students all at similar stages to run them. Moreover, many/most students prefer the flexibility of self-study and there is a wide range of learning materials and formats available.

I recall spending time reading the theory books which I thought were quite good at the time (and still do). The knowledge is mostly relevant and well presented. The questions are often deliberately convoluted to try to catch you out. Not all students respond well to self-study, particularly if they haven’t taken a theory exam for a decade or more. An important part of the preparation involves learning exam technique, such as ruling out the wrong answers first, not spending too much time on one question, answering the easy ones first then return to the more difficult one and revisiting/reviewing all of the questions at least once.

Today there are now many more learning resources, including various online practice question banks which sensibly don’t just give you the correct answer but instead provide text to revise and deduce from. There are also plenty of graphic and video material and online courses. Specialist ground classes aim to finish training and pass the exam in a day (or less).

Although I have instructed many PPL and IR students in the air including pre-flight briefings, I have given relatively little groundschool towards PPL theory exams. Conducting exams will give me much more exposure to how students find them and how best to prepare for them.

Post-exam debrief

I can’t think of any other professional theory exam I have ever taken that has required the invigilator to have any knowledge of the subject. Invigilators are trained in the conduct rather than content of examinations, whether in schools, universities or professional institutions. This applies even to those for the CAA’s own commercial CPL and IR exams, as well as those for all FAA theory exams.

A stated reason for the CAA’s insistence on qualified flight instructors conducting the PPL theory exams is that they can subsequently debrief candidates, discussing questions where they gave the wrong answer. This can be done regardless of whether they passed or not, and I would think the intention is to ensure that any critical misunderstanding is rectified at this stage.

In the past, examiners would have had sight of the examination papers so could see exactly which questions had been answered incorrectly (and what that incorrect choices were). With e-Exams, a knowledge deficiency report is produced (only where at least 50% of questions were answered correctly) indicating which numbered Learning Objectives had incorrect answers. Examiners must then cross-reference this to the Part-FCL PPL Learning Objectives to find the relevant topics. There’s no further information about the specific question unless the student remembers it.

For example, one of our students recently passed the Communications exam with flying colours.

The Knowledge Deficiency Report indicated code 090.01.01.04

Looking this up in the syllabus, this relates to “Definitions – Categories of Messages“. The student thought the question included something about flight safety messages, so I could only surmise that it was asking what type of message this was. I can’t say that I was able to significantly improve the student’s understanding of why he got the question wrong, however I don’t think he will plummet out of the sky or misunderstand ATC due to this specific issue.

Another student passed Air Law with a couple of topics reported deficient including 010.08.01.08 which relates to “Procedures related to emergencies, communication failure and contingencies“. The student couldn’t recall anything we thought might be relevant or incorrect, so again the debrief was not productive.

I’ve not had a failure yet, but each time when I reference the syllabus codes and discuss the topic with the candidate, I find we are both in the dark as to what the incorrect answer might be or the knowledge deficiency involved. Having passed, I doubt the student will go back and study the topic thoroughly and I find myself helpless to make any difference to his understanding.

Compare this with my IR and CPL theory tests where no instructor is involved. As a student, I decoded the Knowledge Deficiency Report myself and revised more thoroughly the whole topic for those Learning Objectives I failed. The exam only probes specific points and failures could indicate a wider lack of understanding of the topic overall.

The procedure under the FAA system differs – students can make one attempt self-study for the PPL exam but if failed must then seek out a qualified instructor to approve and recommend them before retest. Tests are conducted by approved invigilators who need have no knowledge of the subject, but have been trained in exam invigilation.

Advantages of the online system

While I do agree with the longer term CAA migration to online administration online, I’m struggling to come up with many immediate benefits for a student.

1. There is a clear record of test results, easily accessed online which can be used by those who have used multiple flight schools during their training.
2. The quality of the exams is improved because questions are randomly allocated for each student/exam and frequently refreshed and updated.
3. After failing an e-Exam three times, the student is no longer required to visit Gatwick for the fourth attempt. This can be taken at the same ATO/DTO after the organisation submits a training plan.

Downsides include:

1. Added cost, £10 fee to CAA per exam plus setup costs for each DTO (equipment and ground examiner authorisation)
2. Fewer authorised examiners without being able to delegate to trusted office administrators to invigilate.
3. Pre-registration delay, although this does encourage students to register promptly for online services. I don’t really understand why all new CAA Portal registrants aren’t immediately granted access to medical and PPL exam services by default.
4. The revised question bank initially received some positive reviews, although it appears there was a temporary blip when additional questions were added without quality control leading to diligent students failing multiple times. I haven’t seen the questions myself so can’t comment directly.

The main objective from the CAA perspective seems to be around tightening up security to address concerns around lax supervision and widespread availability of the actual questions being asked. It has adopted some of the policies of the FAA system but misses out on others, increasing cost and complexity.

Longer term, the exam results should be available directly to the system that handles PPL licence applications, avoiding the need to print out results and resubmit them, speeding up licence issue. I don’t believe that is the case for PPL exams yet, although it is for CPL and ATPL.

On a more positive note, all of the ground exams I have conducted to date have been passed first time!