Well, IIRC we don't really know what happens to Varys after the death of Joffrey, do we? I think he just disappears, and doesn't re-emerge until the end of Book 5, though I'll admit I don't remember it very well at this point and could be wrong. I thought he re-emerged in King's Landing to facilitate the death of Kevin Lannister, in order to help perpetuate the chaos that could assist in the return of the Targaryen kid.
Of course, I don't believe GRRM imagined he went east with Tyrion, but I suppose the TV show could do almost anything with him at this point.