Use of word "unless"

I was playing a game called "who is faster at sudoku" against my friend and after i won i asked him to play again. He told me to go to hell and my response was "unless we go together" but he told me that it was incorrect. He said that correct form would be "not unless we go togeher". Is it true ?