Both are used with little difference in meaning. Talk to is idiomatic even when the talking is not a one-way affair, in other words, both persons talk. To may of course imply that only one person talks.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.