It is used as a qualifier here. It means that the speaker believes that the man who is not present is responsible (for whatever they are talking about). It hasn't been commonly accepted that he is actually responsible, just that the speaker thinks he is.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
Debpriya DeIs it a sentence interrupter ?I wouldn't say so.