To my ear, the adverb modifies the "made" in the first one and the "retire" in the second one. That is, in the first one, perhaps "they" should not have made him do it. In the second one, perhaps "he" should not have retired .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.