The risk of losing EJ was overblown and once he was drafted, it became Whaley dead set on proving himself right. He traded two firsts to prove EJ was the guy and so we couldn't take a QB high in 2015. He only brought in Orton after Marrone freaked out, Whaley was perfectly content to play out 2014 with EJ, which is why he was the only real QB on the roster all summer. He traded for Cassel, but just like Kolb, that was competition that you could say "they are a little better but EJ has more potential!" And still play him. He signed TT to placate Rex and had serious doubts he could challenge EeeeeJ. Only after TT made a pro bowl, Whaley extended him but that's only because the only thing more important to Doug than EJ is his own job security.
If Whaley wanted Cardale, he'd be out there. He got Rex fired to play EJ for a reason.