Because fewer is the proper word, you need to reformulate the sentence. I suggest: My points were one fewer in every category. Others may have more suggestions.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.