The 1st sentence means you’re turning (only) one light on or off. Let’s say you’ve a bed light and a ceiling one in your bedroom, you Mom would ask you to turn off your lights when you go to sleep.
I usually use light (singular) when referring to a specific lamp with its own switch, and lights (plural) when referring to most other light sources, as when the switch is on the wall, whether the switch controls one or more lights -- because often this is unknown anyway.