"in", because the pan has sides which prevent the oil from spilling out off the sides. The oil is thus enclosed in the pan. This is regarded as more important than the fact that the oil is in contact with the surface of the pan, a situation which normally elicits "on".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.