There are no rules. You have to learn and remember how different words are combined with go. Here are some examples for you to learn.
You go to a place. (goal-directed motion) to go to [the movies / a concert / the supermarket / the post office / school / a party / a restaurant / ...]
You go for the purpose of enjoying some activity involving relative