It's purely a difference in point of view, as explained in another of my recent posts. We hadn't met asks the reader to think in terms of an action that happened in a time period preceding the time period of the main story line. We didn't meet does not.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.