Following some discussion with someone on the subject, it would be natural to say, "In all fairness, wouldn't you etc.?" (I've never heard, "For all fairness."
The implication is that the other party disagrees with you in some respect, and you are pressing your point.