You are correct that 0! = 1 for reasons that are similar to why x^0 = 1. Both are defined that way. But there are reasons for these definitions; they are not arbitrary.
You cannot reason that x^0 = 1 by thinking of the meaning of powers as "repeated multiplications" because you cannot multiply x zero times. Similarly, you cannot reason out 0! just in terms of the mean