To expand on that a little, it's also a highly impractical thing to have or to want.
So sometimes "a pony" is used as an archtype of what you want but can't have.
It's sometimes used jokingly because it's SO common for little girls to express a desire for one ... for example, the single uncle will joke to his brother that he's going to get his niece a pony - and implication is