It's tough to comment because we don't know the specifics.
But I will say, I think it's natural to feel bad. I recently left a senior position and I knew it was the right thing to do and I did not like job I was in. After I told the CEO I was leaving, they tried to get me to stay and I really felt miserable, but a month into the new job, leaving was abolutely the right decision and I barely even remember the old place.
Do what's right for you and what makes you happy.