You say this, but I'm not so sure.
We both know right and wrong, I'm assuming. Your rationalizing is just that. Your friend, I'm afraid, is wrong.
Whether or not I'd choose to defend such a person, especially when they attacked something so close to my heart, such as you with racism and black folks, is immaterial. The choice would still be wrong, indubitably.
And for you to have such a self-righteous attitude about how much you've done for race relations, with such an ugly mark under your belt, is a tough look. The phrase for someone who talks a big game, but chooses letting overt racism slide because it might cause some temporary discomfort is probably a "self-righteous blowhard." You're projecting.
I can say that I'd hope me, and my children wouldn't make the decision you made. Because I'd hope we were better people than that. That we took our principles more seriously than that. And that we knew a wrong decision with a "yeah, but...." is still wrong.