OPB - Funny you should mention a straight route Bakersfield - LA. As early as 90 years ago, Santa Fe did a survey on a route that basically parallels what is today the I-5. They concluded that it was feasible from an engieering standpoint, but there would not be enough traffic to justify the investment.

Keep in mind back then they did not have a sophisticated understanding of geology, there were no environmental concerns to speak of, and NIMBY had not even been thought of. Offhand, I'd say that the number of tunnels required to lay out a feasible route make this a non-starter due to the earthquake hazard - it would just be too expensive to build all those tunnels to withstand a potential 8.0 quake.

Also to be considered in the case of conventional speed rail is that you've got to go where there are people to pick up and drop off. That means places like Palmdale, Lancaster, etc. etc.