'To not have anything to do with someone' is a fixed phrase that means 'to avoid someone and his concerns completely'. Only the negative form is in use.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.