That's right. These tests are required and a railroad's program will be criticized by an FRA inspector if it doesn't show come enforcement actions. Here is the program for certified locomotive engineers. Watch for simliar rules once the conductor certification program comes into effect.

240.303 - Operational monitoring requirements.(a)

Each railroad to which this part applies shall, prior to FRA approval of its program in accordance with ? 240.201, have a program to monitor the conduct of its certified locomotive engineers by performing both operational monitoring observations and by conducting unannounced operating rules compliance tests.

(b) The program shall be conducted so that each locomotive engineer shall be given at least one operational monitoring observation by a qualified supervisor of locomotive engineers in each calendar year.

(c) The program shall be conducted so that each locomotive engineer shall be given at least one unannounced compliance test each calendar year.

(d) The unannounced test program shall:

(1) Test engineer compliance with:

(i) One or more provisions of the railroad's operating rules that require response to signals that display less than a ?clear? aspect, if the railroad operates with a signal system that must comply with part 236 of this chapter;

(ii) One or more provisions of the railroad's operating rules, timetable or other mandatory directives that require affirmative response by the locomotive engineer to less favorable conditions than that which existed prior to initiation of the test; or

(iii) Provisions of the railroad's operating rules, timetable or other mandatory directives the violations of which by engineers were cited by the railroad as the cause of train accidents or train incidents in accident reports filed in compliance with part 225 of this chapter for the preceding year;

(2) Be conducted that so that the administration of these tests is effectively distributed throughout whatever portion of a 24-hour day that the railroad conducts its operations;

(3) Be conducted so that individual tests are administered without prior notice to the locomotive engineer being tested; and

(4) Be conducted so that the results of the test are recorded on the certificate and entered on the record established under ? 240.215 within 30 days of the day the test is administered.