Yes you would need to add 'formal' and drop 'the' to make it read:
"The restaurant is not fancy enough to wear formal dress."
'Dress' does not mean 'a dress'. It means in this sentence and context, clothing more generally. It applies to both males and females. For this reason you would not use the definite article 'the'. You do, however, need to link to the first part of the sent