Thanks for your answer. I completely agree that it's idiomatic, and it does slide off my tongue more nicely. But we do say, "What on earth?" and "What in the world?" which both follow the rules of other uses of world and earth.
Maybe we should just admit that cussing and swearing are immune to, exempt from, and transcendent of grammar.