I would include "had", but it does not seem glaringly wrong without. I prefer to reserve "ago" for use relative to now. In this case I would prefer "a moment earlier".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.