It is a fixed phrase: for that matter = as far as that is concerned; as for that: 'For that matter, you are no better qualified to judge than I'. In your sentence, it means 'as for trusting him'.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.