Hello, Kougo-- and welcome to English Forums. I think you are trying to take 'by the time' too literally in this case. Both point events happened at the same time: I got there and I came too late: By the time I got there, I was too late .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.