To the Left of "Them"

If two people are sitting across from each other in a restaurant, is it okay to say something like "He looked over at the tables to the left of them," even though, technically, the tables are specifically at his left and also at the left side from the perspective of the entrance of the restaurant but actually to the right of the person on the other side of the table who he is sitting with, conversing? That is, is the word "them" okay in that sentence, even though the tables are actually to one person's left (the perspective of whom I'm talking about at that moment) and to the right of the other person?

Thanks.