If it's not a reserved seat, there is little you can do; you should not have left it. However, if you wish to pursue this, say ' Excuse me, but I was sitting there '. In your office, speaking to a colleague rather than a stranger, your remarks are far too rude.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.