The Comma-Before-Too Rule

Is this the rule in deciding when a comma is correctly inserted before a sentence-ending 'too'?

Lou: What are you two doing this weekend?
Chris: I'll be going to a birthday party. I will be attending the book fair, too.
{If two things are involved [here it's the birthday party and the book fair], we use a comma before a sentence-ending 'too', correct?}

BUT:
Pat: I'll be attending the book fair too.
{Pat is simply confirming that he is also attending the book fair, so no comma before 'too', right? He wasn't given the option to attend the birthday party.}

Thank you.