It became more unbearable for me after the AAF and XFL showed us how good it can be. Let the players play, only call the obvious stuff, and have a quick and transparent replay review system.
Keep in mind that I also complain about the refs while watching all the other games the Bills aren't playing in. I feel no shame in complaining about the refs as long as I'm doing it for all teams.
The NFL refs' mantra seems to be "Call the ticky tack stuff, disregard the obvious stuff, get as much screen time as possible, and remember: there's no such thing as bad publicity."