In general leaving at sunset is a good idea but I have been there when there was a lot of railfans and it seemed to be safe. Like the old saying, "there is safety in numbers."

Another fun spot that is more relaxing at night is the old Holt station location on the Santa Fe (by Whisky Slough) I haven't done it in a long time but the last time I was there you could see the block signals from far away change as a train worked it's way toward you.