In the "who(m) to + verb" pattern, "who(m)" is always the object of the verb. "who(m) to go" is therefore incorrect by itself since "go" is not transitive. Instead of a single transitive verb, you can have a "transitive phrase" ending in a preposition, such as "who(m) to go with", "who(m) to go on holiday with", etc.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.