Air Traffic Scenarios Test: Will Modifying the Instructions Change Performance?

The Air Traffic Scenarios Test (ATST) is a low-fidelity radar simulation that is part of the Air Traffic Training and Test (AT-SAT) battery. Past Federal Aviation Administration (FAA) and military research has shown the ATST to be a strong predictor of training success for air traffic control specialists (ATCSs). Because the ATST was such a strong predictor of controller performance, the U.S. Air Force (USAF) was interested in using the ATST as part of its entrance testing selection process for air traffic. However, the current version of the ATST takes about an hour and a half to complete, and is too long for the USAF to use in its present form. In agreement with the FAA, the USAF requested that the FAA shorten (can be completed in less than an hour) the ATST without jeopardizing its reliability and validity. Two studies were conducted to examine whether it would be possible to shorten the ATST. In the preliminary study (Study 1), the authors analyzed data from 391 job applicants that took the AT-SAT during a two-month period. The authors examined scores for efficiency of aircraft movement to destination, safety or maintenance of aircraft separation, and procedural accuracy. Overall, performance changed across trials. Because performance did not consistently increase or decrease across trials, the authors could not simply reduce the number of trials without changing the validity of the test. The authors thought it was possible that, with a more structured approach to designing instructions, performance on the trials might become more consistent, thus allowing the number of trials to be reduced to yield a reduction in total testing time. For Study 2, new interactive instructions were developed. Data were collected from Air Force recruits who were naïve about the ATST. A final sample size of 193 cases was used in the data analysis. Participants spent 6 minutes longer on the modified instructions than on the standard ones; however, both times were substantially less than the currently allotted time of 25 minutes. ATST performance for both groups was poor. Although a profile analysis of the two instruction types showed no significant difference, the multivariate tests for trial revealed a change across trials for the three measures. Additionally, when the authors compared the results of this study with the original sample of ATCS applicants, the authors found large differences in performance that might indicate that the USAF recruits were physically tired and had less motivation to perform well than did other applicants; accordingly, their performance was poorer. In sum, the results of these studies were inconclusive with regard to whether a shorter ATST would be viable as a controller selection instrument. Additional research will be necessary to determine if the modified instructions are beneficial and whether fewer scenarios will maintain similar or improved predictive validity for ATST.


  • English

Media Info

  • Media Type: Digital/other
  • Edition: Final Report
  • Features: Appendices; Figures; References; Tables;
  • Pagination: 24p

Subject/Index Terms

Filing Info

  • Accession Number: 01563637
  • Record Type: Publication
  • Report/Paper Numbers: DOT/FAA/AM-15/7
  • Contract Numbers: AM-B-HRR-523 (task)
  • Files: TRIS, ATRI, USDOT
  • Created Date: May 13 2015 4:40PM