A t-shirt can be called a “tee,” so it makes a sort of sense that a properly fitting garment could be said to “fit to a tee,” but the original expression is actually “fit to a t.” The “t” may refer to a drafting t-square, though that is uncertain.
BUY THE BOOK!