Condemnation requires payment by the govt agency of "fair market value." That of course is always subject to argument, and if it isn't otherwise settled a judge will eventually decide whose estimate of the value is right. UP is certainly not a novice at this kind of thing.

The other consideration in condemnation is that it has to be for a "public purpose." The Supreme Court took a broad view of that in the Kelo case, but many states passed laws both before and after that require a narrower one. So I'm not sure if condemnation of land for building Olympic athlete housing, and sale or rent on the open market afterward, would be considered a "public purpose" adequate to justify the taking. If it were to be used for public housing afterward, with ownership transferred to a suitable public agency, it might be easier to justify (and harder to approve given EIR and public involvement requirements for that). OTOH, there was a statutory total CEQA exemption for the previous LA Olympics; how hard would it be to add a new round to that?

Basic "taking" rules are in the U.S. Constitution. Government absolutely can take private property for public use, but has to pay for it.