My most recent experience with a non hunting user was with a cross country skier who hid behind a tree asking if I see them. I'm wearing bright orange and he was not but I could still hear and see him from a mile away. That doesn't stop them from feeling scared.
I feel like a lot of the fear comes from misunderstanding so a quick conversation during encounters like this go a long way in reassuring people we hunters aren't a careless bunch of hooligans shooting anything that moves.