The rebuilding started at the beginning of Manuel's career not the end. The 2008 team, the year before Manuel's freshman year had one member drafted, 2009 had three, 2010 had three and 2011 four. Bowden's last year was 2009, Manuel's first. Fisher took over in 2010. So first coaching change on 33 years, NCAA violations, no national championship in ten years and a losing record against your biggest rivals. Yeah they weren't rebuilding.
I agree that we have not seen enough to make a judgement. I don't know of he is the guy either, but I don't understand people looking at Manuel's college career and saying he was unsuccessful or mediocre.