You should use preposition "in" in both cases. Using "with" implies that the subject in this sentence will be holding a sweater / a shirt, not wearing.
Could I ask you to give me your opinion on this thread, I don't know whether to use of, to or from? I've gotten different answers from lots of people, so I'm not sure.
1 Walk in the opposite direction of/to/from the bus stop.
2 Walk in the direction opposite to/of/from the bus stop.