Also remember: those Youtubes of "worst track ever" are shot with a long telephoto lens, which compresses everything and exaggerates all of the movement. Probably sped the video up a bit too. Even on rough track, trains operated at a proper speed (sometimes well below 10 mph) don't bounce all over like a fighting kite in a stiff and shifting breeze. They move around plenty, but not that way.

If it's not a well-maintained main line, it's a good idea not to stop close to the tracks at a grade crossing. The white stop line painted on the street is the minimum clearance, not the maximum. If a train is moving slowly for no other apparent reason, especially if you see wheels dropping at worn rail joints or cars rocking back and forth a lot, back up, and if you can, back up farther. Grade crossings are popular spots for wheels to jump the track if things aren't in great shape.

We have a section of main road in town that's closely paralleled by an out-of-service line. Out of service doesn't mean abandoned. A local heritage group runs a train through there occasionally (yes, properly flagged), positioning for tourist runs farther down the line, and gives the track a little maintenance. The crossings have no limit lines or other road markings. I still stop about a Fiat's length back from the tracks (which others on the street don't always appreciate, but the signal loops are set up for that so no foul), just to keep in practice.