'Be going + ADVERB + with/for', MTL-- those are all adverbs.
Bad things happen to us, but the 'going...for/with' collocation reflects in concept the events/experiences we are 'passing through' rather than those that are 'acting upon' us. That is why I see 'to' as inappropriate. Because idioms have a mind of their own, I see no grammatical reason why 'to' is not used here.