He needs to figure out a polite way to end the visits when they become unpleasant for him. Until he can think of such a polite way to do so, he simply won't visit at all.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.