I do care a little bit. I also used to frequent r/hpmor, but now I don't really have the time for it. And yeah, HPMoR is a clever way for him to spread his message. I think my criticism boils down to EY himself seeming a bit.. off. Based on a Google Hangout interview I saw he seems to be the kind of person who talks at people, not to them.
I interact with him on facebook and reddit a lot. Have read a lot of the Sequences, etc. Maybe I identify with him because of a similar background (I once read a small autobiography of his, written when he was quite young, that would definitely rub anyone the wrong way -- he's basically claiming that he was so much smarter than everyone else growing up, incl. his parents, that his childhood was irreparably fucked -- thing is, it's probably true) and I can to a lesser extent understand where he's coming from.^ Anyway, a lot of the hate for him stems from simple anti-intellectualism, some of it from anti-futurism, and some of it from, like you say, just a sort of feeling you get when talking to him. He's not arrogant exactly, but there isn't an ounce of false modesty. As a society, we've come to heavily weight our expectations of extraordinarily smart people toward humility and false modesty -- but in Yudkowsky's view, that's just a misrepresentation of the truth. ^my counterargument: if you're really that smart, you can read people like books and learn to get along with them. Not to do so is arrogance, laziness, or a sign of some slight mental imbalance.
Yeah, he means well and seems mostly honest but is probably hampered by some neuropsychiatrical disability. I think I traded a few words with him, but not many tbh. I had a bit of a difficult childhood too and have a hard time reading people, but I've stubbornly taught myself how to be better at it.