The problem you have, as written, is that it can be understood in two ways. Perhaps you are saying we lost by scoring two points to their one point, or they scored two points for every one point that our team scored, so the score could be two to one, or maybe six to three, depending. Throwing in the "two points" or "two goals" makes it read better because you avoid the "two to one" which is apt to be read as a ratio, two points for every one point.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.