Remember that lacrosse started in the Lower Ontario, WNY region as an alternative to war for indigenous peoples. It is a sport heavy in tradition in this region. While the modern versions of lacrosse have spread across New England and the collegiate space, we shouldn't forget it's history.
There are some rabid fans that have been with the team since their inception.
The atmosphere in the building during games is non-stop. Blasting music while game is in play, a constant moderation over the speakers starting chants (what's he got? Nothing!), the B-O-X chant, and the heavy physical play on the field usually resulting in a fight or two each game. It's just.... different.
Your right, most won't. But some will. As a fan I'll always remember the 3 Kilgore brothers, Johnny T, etc. The building is shared by two franchises (the Bandits even have their own nice locker room). They deserve to have their banners up in the rafters too.