That might be one case. There is also the case in which 'speak with' simply indicates a longer conversation than 'speak to'. With such common verb phrases, there is seldom a simple mutually exclusive bifurcation (that's not the word I want, but what I mean is that there is no simple two-dimensional distinction of meanings).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.