Certification of airport security officers using multiple-choice tests: A pilot study

Initial and recurrent certification of aviation security officers (screeners) is mandatory in Europe as specified in the EU regulation 185/2010 and in national regulations. The certification includes a practical test, an X-ray image interpretation test as well as a theoretical test. This paper focuses on theoretical tests which are used to evaluate whether screeners have the relevant knowledge about regulations, security control procedures, etc. To test this knowledge multiple choice (MC) tests can be an efficient, reliable, valid and standardized method. The development of a MC test does not only require expertise in aviation security but also expertise in test development and psychometrics. Besides the standardized development of MC questions, the scoring method is likewise important. For this pilot study a first version of 188 MC questions from several EU Member States were evaluated with 183 screeners. The results revealed that it is of utmost importance that the development of MC questions is based on MC guidelines, i.e. apply expertise in test development and psychometrics in order to guarantee a high quality. Besides the test development the scoring method has a substantial influence on test difficulty, reliability, item difficulty and item discrimination. Implications of the three different scoring methods were discussed.