No, it's not correct. It contradicts itself. "had gone" indicates "at some time before the reference time (already in the past)" "ago" indicates "at a certain time before the reference time (which is always the present)" Either "had gone" or "ago" must be changed to restore harmony!
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.