I think that the replay rules are silly. If you challenge, and your challenge is successful, why should that be held against you?
Refs are afraid to call what they see on the field, knowing that the challenge system is in place to overturn some of their egregious calls or missed calls. However, by making or not making a call, the referees' call or non-call is presumed correct, with the threshold for overturning via replay (assuming that (1) the play is reviewable, and (2) a team has the ability to challenge the call) being quite high.