Pronunciation-wise, there is no difference between father's and fathers', and so even when you think you have said fathers', people will usually interpret that to mean father's, as it is the norm for people to have just one father.
If you did that by writing, it would have been a mistake.
Correct:
My father's name is John.