The combination of "noun + to + verb" in such costructions often has a sense of futurity and/or tedious repetition about it.
my punishment is to peel potatoes (futurity and tedious repetition)
my lot is to be the father of a teenage ruffian
my wish is to be a star (futurity)
That's why your example doesn't work with "hobby". A hobby, though being repetitive, is often pre