I'd go with D. "Had gone there" suggests that someone had already left for the game but he was still in the place where the football game took place. "Had been there" implies he had already returned from the game.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.