Up until 5 years ago, they didn't have the large fanbase. That came with winning, once Brady took over at QB.
Frankly, the Boston fan knows very little football outside of their own football team because the Pats were always a distant 3rd and maybe even 4th behind the Red Sox, Celtics and Bruins.
Many bandwagon fans/frontrunners have jumped on board in the last 5 years, who never cared about the team before.