The first is correct in most contexts. 'Dinner' here is the 'institution', the meal that we have every night. The first is possible if we are referring to a particular dinner that has been arranged.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.