Is it ok to say that you do not inhabit the same world (or that you are not on the same planet) with your wife and with your mistress.Meaning that you live two separate lives with each of them.
"He's from another planet." is quite common, too. As far as I understand this idiom, it's more about being different (being of a different kind/style) rather than living with mental distance or not being close to each other.
Maybe "(the worlds of) his separate lifes have nothing in common"? Or "his two separate lifes could easily take place on different planets"?
Instead of law and order, you suddenly live in lawlessness and disorder, in an intimate, extramarital affair during which you comfort yourself by saying that you are not cheating anybody because you do not inhabit the same world with your wife and mistress.