I don't think there's any fixed rule. The "ing" is more common, but as in you're example, both are possible. I can't think of an activity where the infinitive is preferred.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.