It is merely a matter of speaker's and/or listener's viewpoint. If the speaker is envisioning the action as occurring in his direction or (for politeness's sake) the listener's direction, he will probably use 'come'. If he envisions the action as moving away from either or both of them, he will probably use 'go'.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.