Salam is basically right, but it is more than just arms or armrests. Some straight-backed dining chairs also have armrests, but are still rather skeletal, so we sit 'on' them. The difference is really quite tactile: if you physically sink down into the (usually well-padded or -stuffed) armchair, then use 'in'; if you perceive yourself as more perched on a chair's surface, use 'on'.