It depends on how you are thinking about it -- whether you're thinking about it more in terms of leaving where you are (I wish I could go), or moving towards your friends (I wish I could come). (A native speaker would not stop to analyze it like this. He would just pick one or the other, and if you asked him why, he would probably have no idea.)
I think we use come to describe movement to the place where the speaker or the hearer is, and use go to describe movement away from the place or position where the speaker or the hearer is.
So I think you should say that: I wish I can come, but I've got work tomorrow.