No, you cannot remove the last "to". Original reply is incorrect: he did not understand the meaning of the sentance. "having you all to come back to" means "being able to come back to you all". The same "to" is underlined. The expression is "come back to" something, meaning "reyturn to" something.