There is no necessary accounting for mathematical terminology, any more than there is for that of any other English for Special Purposes, in terms of accepted grammar. However, I would suggest that we are not talking about numbers in the normal sense (like 4 thousand soldiers), but are speaking of what is called the 'thousands' position (the fourth from the right) in the decimal number system-- that is, the position where thousands appear, whatever the quantity we have of them (3, 4, 5, or whatever). Does that help?
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.