The first time .... + past tense

Sometimes this construction can be really confusing for me as I can't seem to spot a consistent pattern in terms of what tense is appropriate. To me, it seems as though the word order itself has a huge impact on it. For example:

The first time I went to Spain was in 2013. I have no issue using the past simple when the first time is at the beginning and I also think the past perfect sounds kind of odd here.

That was the first time I had gone to Spain. However, in this case, it's the complete opposite in that I feel a lot more comfortable using the past perfect and the past simple sounds worse. Naturally, I don't trust my intuition as I'm not a native speaker, so that's why I'm here.

Some other examples:

The first time I ever went fishing was 20 years ago.
It was the first time I had gone fishing.

Also, the past simple seems to work best with questions as well. Using the past perfect here sounds strange to me.
When was the first time you went to Spain?
What was the first movie you ever saw?

What's going on here? Can someone explain? Thank you.