I think no one has helped because no one can understand it. There is not enough context to explain the relationship (if any) between the adoption and a doctor's office.
"I never think about the fact that my daughter was adopted, except in the doctor's office, where I remember, and make sure to tell the doctor, because family history can be important when considering risk factors and diagnosing certain diseases."
"I never think of it outside [situation where I think of it]" is equivalent to "I never think of it except when [a situation where I think of it obtains]".