Yes, it's fine. 'Rags' also suggests his clothes may be torn. If you tear an old shirt into pieces for cleaning your floor, you are tearing it into rags.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.