There is no real difference between the two. The second emphasises that you had dinner first slightly more than the first. No you do not have to use the past perfect for this sentence.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.