Hi Jin_H, What you would say would depend on why you didn't want to tell him off. If it's because you didn't want to upset him, or stop him enjoying himself, then you could say any of: I don't want to spoil the party. I don't want to be a party pooper.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.