Past perfect or past simple?

Could anyone explain the meaning and why this is correct/incorrect (taken from an English course book answer key):

I went to Japan in 2004 and 2016. SECOND

In 2016 it was the second time I had gone to Japan.

Does that mean I went to Japan after 2016 once again? Could use the past simple here (I went) and would the meaning stay the same for a native English speaker?