The railroad structure (tracks, ties, etc) and the rights to operate are owed by the P&W (G&W), the property under the structure is owned by Oregon Department Of Transportation (ODOT).
I'm very sure the Port of Astoria will not allow to rails to be removed. I'm sure they would pursue purchase/donation, or some other form of transaction, if it came to G&W wanting out of the route.