It depends, is it going to be a discussion or a one-sided dialogue?
"With me" implies both parties will be speaking/discussing. "To me" implies you just want the person to speak to you and you won't necessarily say anything.
It also depends on context. If I am your parent and you want permission to go somewhere, I might tell you, "You have to speak with me first." If you and