Last point first: Because Caltrain is running on industry frequency, there is no need for a converter, and not even for a phase adjustment device, because they would run in phase with the grid. So there is always enough capacity to take up a couple of MW of regenerative braking energy.

Because regenerative braking is not the only system (you state correctly, it is not fail-safe), there is always a backup system around (conventional air brakes).

Do I assume correctly, that those 250'000 pound is the maximum braking force? If this is also the maximum tractive force, there is no issue, even with the best electronics, you won't get beyond that force when braking. And, as stated elsewhere in this discussion, when you are electrified, you do not run underpowered overweight trains; you run faster, lighter and more frequent trains. Keep in mind that modern locomotives can maintain the maximum tractive force up into the 50 to 80 km/h range.