If you went there for advice/treatment then you would normally say "I went to see the doctor yesterday" or "I went to the doctor's yesterday". However, "I was at the doctor's yesterday" does not exclude the possibility that you went there for advice/treatment. You might, for example, say this if you wanted to explain your whereabouts yesterday.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.