While it's true that, theoretically, the "I've" may be the element assumed to be retained in the second clause, I think your first impulse to say that the second clause is in the simple past is closer to the truth. Tenses are mixed quite often. There is not always a satisfying explanation for every possible combination in every possible context.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
So, which is better, American or British English?