"for all" vs "with all"

Would you tell me why "with" is inappropriate in the sentence below?

There is much truth in what he says, but, ( ) all that, the problem under consideration is not so simple.

a. in b. for c. at d. with

The textbook says that the answer is b. for.

I know "for all" means "in spite of / despite", but Oxford Advanced Learner's Dictionary says "with" has the same meaning "in spite of / despite" as below.

With all her faults I still love her. [O.A.L.D.]

For all its clarity of style, the book is not easy reading. [O.A.L.D.]

In these sentences, "With all" and "For all" seem to be interchangeable.

Then, again, why is it wrong to say "There is much truth in what he says, but, with all that, the problem under consideration is not so simple."?