It doesn't have to mean 'for me'; in other contexts it could mean 'for him/her/them/you'. The meaning is 'to me he does not seem to be a burden at all; I am happy to bear him'.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.