I used to be in your camp, eball, and I'm about the same age as you. But as life goes on I have come to the decision to not take myself so seriously. Especially on gamedays at the stadium.
Not saying that's your issue, just that it was mine and I'm now past that. I live a pretty buttoned up day-to-day life in the professional world, so wearing a jersey is a pretty tame way to relax a little and be a kid again.