Sounds like the same problem I had, yellowjackets. From my research on them (trying to find the cheapest way out of course) the queen starts a new colony and they start appearing slowly in June. They can reach numbers around 4-5 thousand by fall. They'll all die off in the fall, with only the new queens overwintering and she'll find a new site next spring.
I could stand by the hole and they would fly right by. Anytime the lawnmower got close though they got a little angry so we had a guy come spray for $125. He missed the nest the first time and they found a new way out around the dust he applied. Came back for free and got them. Whatever you do don't seal the hole, they'll find a way through the house if they need to.