He was an offensive lineman as a player. And he coached tight ends and offensive linemen before becoming an OC in New Orleans. His experience lies on the offensive side of the ball. Can't see how that's even debatable.
However, I do agree with you that he's not an offensive specialist. A specialist implies some level of competence.
As for your second point, if he's allowing the three phases of the game to be run by his coordinators, what exactly is his job during the game? Other than deciding to punt on 4th and inches or flipping a coin to decide whether he should use a timeout.