Generally, you are correct, but time isn't everything in such constructions. The degree of doubt might determine which of the two tenses, the past perfect (for most doubt), or the past (for less) should be used, irrelevant of the temporal element involved. Should you have the opportunity, go to a good university library and consult this great reference (it's out of print now).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.