I have never had a problem at a Bills game but I'm also always "on-guard." I don't live there anymore but when I go back, I'd love to take my daughter. Odds are that I won't, for a long time, because people are such dickheads. I have been to Eagles games and find that in the new stadium, the behavior is better than at the Vet.
I wouldn't wear another team's colors in either place. It's one thing for people to give you a little razzing; it's an entirely different thing to throw things, curse, etc. If you ever want to feel what a good atmosphere is, go to a big-time NCAA football game. The tailgating is great fun--great CIVILIZED fun. The atmosphere inside is as electric as any Bills game. And somehow, people get along without being knuckledragging morons.