Your sentence seems correct, except that you can throw away the last 'to'. However I would like to ask: Do you mean to say that if he is not coming, you will prefer to go somewhere else? For example if he is not going to eat in this restaurant, you will prefer to eat at another restaurant?
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.