I was not aware of the derogatory nature of the term in the UK—it sounds like it’s akin there to “R” word here, a word that immediately tells me the speaker using it is a jerk.
I grew up in the ‘70’s and ‘80’s—back then it was slightly derogatory, but not offensively so.
All that said, Brady has actually done lots and lots of charity work for years with the mentally and physically handicapped (it’s one of the few things I’ve admired him for) and I’m sure meant nothing derogatory by it. He’d probably be mortified that it could be taken this way.
I think I just threw up in my mouth a little having to defend Tom Brady…..