"that far" refers to the very small distance that he is indicating with his thumb and finger. In other words, they are extremely close to being chucked out.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.