Basically, the writer is saying the people he admires or likes shares different values/empathies (There isn't good connection between them or they can't find a common ground for conversations - don't get along all that well).
However, the people he loves - he accepts all their flaws and strengths and they make him feel at home. (They have similar tastes and interests, they make each other