They would both work, but I would normally say The man in the blue shirt when referring to the clothes he was wearing. If it were something he was wearing that is not clothing, such as a backpack, I would use with .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.