"He said that he had met her three years before" is what you want. "ago" is always measured from the present moment. To change it to the indirect form , you need "before" in place of "ago", because the "had met" is no longer connected to the present moment.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.