Question: is the intent of the sentence "The men who were having a discussion did not seem very friendly" or "The men, with whom we were having a discussion, did not seem very friendly"? If the first, then with should not be added to the sentence, as it's a conjunction and there is no "other" to link the men to. And that's that.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.