"swim in the same social ponds" - this is a creative rendering of the expression "move in the same circles," which means the same social group. ; even if they were not close friends, they would have friends in common. You should know this expression can be used in a straightforward way, but (as here) it is often used ironically by someone of a "lower" social group speaking of someone more prominent.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.