It refers to an imagined situation in which he did not have to ask. You could argue that he preferred in the actual situation, not in an imagined situation, but in practice saying "preferred not to" (no "would have") implies that someone didn't do the thing described.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.