How maddening it must be to him to be upbraided

[Holmes said] “Ah, thereby hangs a rather painful tale. This fellow is madly, insanely, in love with her, but some two years ago, when he was only a lad, and before he really knew her, for she had been away five years at a boarding- school, what does the idiot do but get into the clutches of a barmaid in Bristol and marry her at a registry office? No one knows a word of the matter, but you can imagine how maddening it must be to him to be upbraided for not doing what he would give his very eyes to do, but what he knows to be absolutely impossible.”

The Boscombe Valley Mystery, short story

Hi. Does the red part mean scolding him must drive him mad?

Thank you.