I personally find "where is the nearest office to me" jarring, as "to" indicates direction toward the object of the preposition. I think it sounds more natural to say "where is the nearest office from me (where I am)?" instead.
After talking it over with a friend, I think "to" might be better than "from".
His argument was " 'close to' and 'far from' ".
But, I still think using "to me" forces you to take the perspective of the "nearest office" in this case, due to the directional nature of 'to', and it sounds weird to my ears.