A negative (not) in the simple past tense with a verb that shows an action (go) indicates that an event did not happen. There was never a completion of the event -- no arrival at the doctor's. If an event was never completed, there is no time at which it happened. So the sentence says, There was no time at which she went to the doctor's and arrived there, and at tha