It is not that it is a small or large space-- it is that it appears in the mind's eye as a point reference or as a 2- or 3-dimensional space. If the speaker is thinking of the vast city, he says I arrived in New York ; if he is thinking of a point on the map, as opposed to other cities, he says I arrived at New York . You can work at or in a gas station : it is a matter of whether you are viewing it as a point, a place along the highway, or as a structure and you're within it.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.