But the other person whom I'm planning to visit is in another city. So shouldn't go be more appropriate?
I assume you mean you are talking on the phone. We still normally use 'come' in this case. Our point of reference is the place where the other person is.
eg I'm in the living room, and my wife is in the kitchen. She calls my name.