Context would tell you.
If person A had just said "I don't like like, and I don't visit him," then if person B says "I don't like him, and I don't visit him, either," the B is agreeing with A on both the "not liking" and the "not visiting."
If person A simply says "I don't like him, and I don't visit him either," then A is just making a statement about two things.