Your observations seem accurate to me. All of the options you cite are correct and "acceptable," and I agree with your choices in structures where one form is more natural than the other. I don't know of any rules, but I'm sure a diligent search would turn something up.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
AnonymousI recently had a student ask me when a gerund or an infinitive should be used