You could say We are going to go without him , (which often implies that we don't want him to go anyway) or you could say We're all going, except for him .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.