Both are correct. With is more polite, as they are part of the conversation. To makes it sound like you wish to give the other party instructions, but is commonly said.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.