I think I may have suggested this website before: http://www.googlefight.com/ It's a great way to check which expressions occur more commonly. Basically, it compares how often words or expressions appear on the internet. Type in "none of your affair" and "none of your affairs" and you can see which is more common (
"It's none of your business" is the more natural expression. The word "affair" is only ever used in modern speech for an adulterous relationship; as in the following:
"While visiting France, she had an affair with a married man."