I believe "matters" has a much broader definition and can refer to any subject of concern or feeling or of emotional importance. I take "affairs" to have a romantic or sexual connotation, or perhaps referring to one's troubled personal business of any kind, but not emotion in general. This is just my opinion, however.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.