"to be" or "is" ?

I'll admit that I was tempted to write "to be or not to be" in the subject box for this one but thought I'd avoid the cliché.

At any rate, I've got a quick question, and I'll have to "set the stage," so to speak, as I think the given scenario might change or influence one's answer:

If I were speaking with someone and we're sort of at odds over a situation, say, a party that I didn't attend, and we had this conversation:

Him: "I know the reason why you didn't go to the party."

Me: "Oh yeah? And what would you imagine that reason to be?"

Is "to be" correct here? Should it be -- or can it be -- "is" instead? "And what would you imagine that reason is?" Is there a difference in meaning? Thanks in advance!