The problem with "slim-fit" is that it's a noun (though dictionaries call it an adjective). Nouns are not (usually) modified by adverbs like "most", which is why it sounds wrong and is best avoided. "Most fitted", where the adverb "most" modifies the adjective "fitted", is fine and natural.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.