Depends on the contract. I know ours says (said, since we got the house), we had a working relationship with our agent for 90 days...but could sever the relationship before then if we were dissatisfied.
Clutch, your business arrangement with your agent is just that: a business arrangement. It can be ended. If there's a written agreement between the two of you, and you can document his non-performance, you can kick him to the curb relatively easily. If everything was done with a smile and handshake, it'll be a lot harder (he can take you to arbitration, and although you'll still probably win, it'll piss you off)...but you can still kick him to the curb.
Either way, if you're dissatisfied, ditch him. You're not required to put up with unsatisfactory service. Unless it's a cell phone company.