"Medium" is correct, but sounds unnatural to me. I would use "means." The sentence order is a little awkward.** I'd suggest, "So our only means of communication was English." Or, "So our only means of communication/communicating was to speak/write in English." Or, "So English was our only means of communication."
**(To my ear, "only" appears to modify the verb "was"