cbakumar,
You are correct.
Who - subject.
Whom - object.
I went to the market with them. Them is an object.
Your sentence takes the same form except, you moved have replaced "them"with "whom" and moved it toward the front of the phrase.
...,with whom (except your mother) I went to the market.
See number 5 in this articl