I think both. If you search "irritating for people" and "irritating to people" on google both combinations are used. In the american corpus it shows that: "irritating to" is used a lot more than "for" this could introduce a noun, pronoun or be the start of an infinitive.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.