Yes, it can be used literally too, in reference to clothing fitting. In fact, with the word "on" included, I would assume that. I am from the UK, so my usage seems consistent with the usage notes on that page.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.