I take train photos too, so I'm equally guilty of doing what you were doing. We also need to remember what each one of us do for a living. I just might have a problem with somebody visiting my office, sitting at a chair next to my desk, and snapping photos of me at work. (Dang, that would be boring). We need to remember that it's not just "innocent" railfans that take photos or railroaders at work (whether our lens is pointed at a person as the object of the photo or not)--there are people out there who, by capturing an image of a train moving without some aspect of safety equipment functioning, are possibly there for the sole purpose of getting somebody in trouble. From personal experience, most crews don't have a problem, but some think of us as pests, and some as the poperatzi--hoping to get a shot that might have consequences. If somebody doesn't like what you are doing, it's probably best to respect their request to move on (unless you'd like a cameraman watch you work). You did the best thing possible in that situation. What we don't need is for some of "us" to start mouthing back at these guys because we have a "right" to watch them work.

[Not trying to start a flame war, just tossing in my $0.02.]