Broken often refers to something that has an actual break that causes is not to function, such as a lamp or a lock. Broken down is used to describe a machine or appliance that is no longer working, such as a car or a washing machine. I don't think I'd use it with items of clothing, which are probably worn out or torn .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.