Told me to do so

Hi. Is the underlined part correct? Thank you in advance for your help.

(Two friends talking)

A: Let's take things from other people here.

B: I won't.

A: Why not?

B: Because my dad told me not to do so (How about change "so" to "that"?).