Both sentences are okay. The one with "go to" would imply much more interest and enthusiasm on the part of the speaker - the speaker is really interested in where you went on holiday. The addition of the word "to" enhances the rhythm of the sentence and makes it more lively-sounding.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.