From the full context, I see that the man in question is waiting for someone to meet him. "call time on him sharp" means "give up waiting promptly at (or very shortly after) the agreed time of the meeting".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.