Brewer's Phrase and Fable says it's from the T-square used by carpenters and such to make things square. Things "fit to a T". Very often we can't know how sayings arose, but Brewer's explanation doesn't seem too far fetched.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.