Come is more natural.
I'd use go when I'm saying 'you want to go with him/her' for example - He's going to Paris this Thursday, you want to go with him?
When I'm talking of myself, I'd rather stick to 'Come'. It'd mean I'm going, you want to come? However, if I say "I'm going, you want to go", It'd be a little awkward.
However, that's my opinion; I hope it helps.