There's no rule. You'll just have to learn them, I think. These are the ones that spring to mind, but there are probably others: I went to school BUT I went downtown/uptown.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.