I have always presumed it was from taking a shortcut across someone's garden rather than walking all the way to the corner and then turning right (or left).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.