It was my mother's is the only option. It means that 'it' (whatever that may be) belonged to my mother.
It was my mothers' makes no sense because the apostrophe after the letter 's' means that 'it' belongs to more than one of your mothers, (unless of course you actually have more than one mother!).