I interact with him on facebook and reddit a lot. Have read a lot of the Sequences, etc. Maybe I identify with him because of a similar background (I once read a small autobiography of his, written when he was quite young, that would definitely rub anyone the wrong way -- he's basically claiming that he was so much smarter than everyone else growing up, incl. his parents, that his childhood was irreparably fucked -- thing is, it's probably true) and I can to a lesser extent understand where he's coming from.^

Anyway, a lot of the hate for him stems from simple anti-intellectualism, some of it from anti-futurism, and some of it from, like you say, just a sort of feeling you get when talking to him. He's not arrogant exactly, but there isn't an ounce of false modesty. As a society, we've come to heavily weight our expectations of extraordinarily smart people toward humility and false modesty -- but in Yudkowsky's view, that's just a misrepresentation of the truth.

^my counterargument: if you're really that smart, you can read people like books and learn to get along with them. Not to do so is arrogance, laziness, or a sign of some slight mental imbalance.