The complainers do have some valid points. In order to have speeds appropriate to the "HS" designation, HSR is similar to a freeway in visual and community impact. So, while at-grade might be acceptable out in the country it isn't where anybody lives - which is most of California outside of the south-central San Joaquin Valley. And even in the SJV, there are towns.

Most major freeways were built in the 1950s-60s when there were a lot fewer people around. Community impact-wise, we've been paying for the "rural" designs ever since. In town, you need sound walls (ugly, but effective for nearby properties for near-grade and elevated sections), full grade separation above or below, not at-grade with streets over (there's a reason Alameda Corridor is in a trench through the populated areas), and generally out of sight where possible (subway/tunnel). There have been several projects to put freeways in tunnels where they were originally just depressed, and a few have actually happened; all that were done have been more or less successful at reconnecting at least part of a community. I would expect people to insist on a true HS line in LA (not just a modest upgrade of Metrolink) meeting the same standards a new urban freeway would in terms of community impact (see: the 710 "Gap Closure" saga, "Big Dig" in Boston, etc.), and savvy lawyers would probably make that point to FRA under NEPA even if CEQA is dodged.

Ultimately, however, no matter what you do some eggs will be broken. This won't be a Negative Declaration or FONSI. So money will be needed to avoid, minimize, and mitigate.

As with the HSR-via-Caltrain fiasco, either HSR will have to give up on actual high speed in LA and mosey into town at 79 or so on Metrolink, or things will cost a lot more than current estimates. Giving up on the urban lines at both ends will probably require giving up on the 2 1/2 hour timing end/end - would need 300 mph in the Valley to make up for delays in the mountains (you're not doing 200mph on 4% grades, even on a direct Tejon route) and urban areas, which nobody is doing on steel rails in regular service (think maglev or hyperloops instead). A 3-3 1/2 hour run (more feasible, given the limits) is still good, competitive with other HSR services, and competitive with real-world driving and flying times, but won't meet the political mandate of The Proposition. So HSR needs to have a Good line at least to Burbank. Good Luck.

PS: thinking on the mountains, how about a nice disaster movie plot? Over the top at 200 passing Gorman, then losing brakes entering the downhill tunnel. Could you hit the sound barrier coming out of the tunnel, and what would be the vertical Gs at the bottom on the pullout? Could you make it to Fresno or downtown LA without touching the throttle? Sound walls probably wouldn't be very effective for a transonic source ...