I’m going home

Bruce was an Australian and worked for a newspaper in Sydney. Then he thought, “I want to see Europe, so I’ll go to England and work for a newspaper there for a few years.” He flew to London and soon got work, because he was good at his job. He lived in a small but comfortable house near London, and he had a small garden. He enjoyed working in it on Saturdays and Sundays. He had nice neighbours on both sides, and they often worked in their gardens on Saturdays and Sundays too, and then they talked and joked together.
One day he was digging a hole in his garden to plant a bush when one of these neighbours came to the fence between the two gardens and looked at Bruce’s work. He laughed and said, “Are you making a swimming-pool?”
“Oh, no,” answered Bruce, “I’m going home.”

I don't understand this joke. Could you tell me why the speaker said he was going home?