As a teenager, I lived just a couple hundred feet from the 4th Sub (aka Surf Line) in El Toro in the late 70s/early 80s, and it was indeed as you describe. Me and a few friends got to know the special agents who patrolled there. We called one "Maverick" and the other "LTD" for the cars they drove down those weedy, rollercoaster access roads in the pre-SUV days. More than once we phoned them whenever we came across junk that someone had left on the track. Naturally, we took the time to clear the mess rather than wait for the agents to get there before the next train. Sometimes it was wooden pallets stacked five or six feet high, other times shopping cars filled with rocks and cinder blocks. Thank goodness none of it ever derailed a San Diegan flying through there at 90mph.