Grammar check

When clarifying that "we" (referring to the two of us who were just now speakiing on the phone.) The person I am addressing, thought when I wrote, "We were just on the phone." Meant I was on the phone with another person. Theefore I want to clarify, that when I wrote "we" I was referring to the two of us. Should I say, "we" who were speaking on the phone" meant "you and I" or should I say, I was referring to "you and me?"