IMHO the difference would be that with "couldn't" you have estimated the "true" time based on factors or calculations relating directly to the current instance. ) With "shouldn't" you have estimated the "true" time based on more general factors, or experience derived from prior instances. That is, we had an opinion about how long it ought to take before we began.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.