We don't use "wear down" to talk about clothes, only "wear out". ) When used about people, "wear out" tends to be more physical and "wear down" more mental, though this is not completely clear cut. For example: The kids are worn out from all that running around.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.