Meaning

Hello Friends,

Would you please tell me about the bold part?

“Between you and your father and your brother,” Alexandra said, “that rug is a disgrace. Hank, when I came to keep house for him the first thing I did was have it dyed as dark as I could. You remember how it used to look? Why, there was a black path from take out. . . .”
Hank said, “I remember it, ma’am. I’m afraid I was a contributor to it.”
Jean Louise drove the putter home beside the fire tongs, gathered up the golf balls, and threw them at the spittoon. She sat on the sofa and watched Hank retrieve the strays. I never tire of watching him move, she thought.

My Interpretation: he was recovering the memory. but I'm wondering maybe he was gathering the balls.

Thank you.