'On' his more usual, but many people use 'in'. A shelf is a 2-dimensional plane, but a bookshelf, if it is a unit containing several shelves, has a 3-dimensional aspect.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.