He is accusing you of being a bit impolite in telling him that he is wrong (usually for some behaviour), and he is saying that it not right for you do do so.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.