Racism does not hurt white people. Sexism does not affect men. Ableism does not hurt the abled. Homophobia does not have an inverse. If you argue against or play devil's advocate on any of these points, you are Wrong. Historically, statistically, factually, empirically Wrong. It is as simple as that.

I'd argue the opposite: I think racism hurts white people, sexism affects men, ableism hurts the abled, and homophobia hurts homophobes. I do not think the effects of oppression on the oppressor are comparable at all, but I do think bigotry has a powerful stunting effect of the emotional capacity and overall humanity of the oppressor. In my experience, I've found that the more I widen the scope of my empathy, the more I get out of life. I also believe this generalizes on a much greater scale, but unfortunately I don't have a very strong argument for that.

You can look at it from a more practical perspective, though. For example, by disenfranchising women you lose about half of the able minds on the planet. Though that has an absolutely disastrous effect on women, it has an indirect negative effect on men. Of course, I wouldn't make that my primary argument for treating women like human beings, but I do think it's sound.