There is different between them. "Had been" indicated the complete thing. "Had gone" meant that you were still at college in your country when you said this was not my first time here at college.
He had been to Hong Kong.
He had gone to Hong Kong.
Above sentences are different. The first sentence meant you complet