Time expression changes of third conditional in reported speech

Hello,

I'd like to ask that whether we should change the time expression when reporting third conditionals. As we keep the tense in both clauses when we report a third conditional sentence, we just change the pronoun. Do we also change the time expression?

For example:

Tom: “If it’d rained yesterday, we wouldn’t have stayed at home.”

Tom said that if it’d rained yesterday (should be the previous day or not ???), they wouldn’t have stayed at home.

Thanks a lot.