I would change the structure to: It's irritating to people to play against him. This makes the situation irritating, rather than the person , and I think that's what you intend here. Of course, the person in question could be irritating himself, but I think the irritation you are talking about here is really the situation.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.