If you're talking about placing it in it's receptacle, and the receptacle is tubular, I'd say "Insert the battery into the device."
I wouldn't exactly call it incorrect to say, "Insert the battery in the device."
Otherwise, I'd say "Install it in the device."