It's a long way [to have to come down] to~

The protagonist, Philip moved in with his uncle Mr. Carey, the Vicar of Blackstable after his mother's death.
He goes to the medical school St. Luke's in London.
He came across his ex-girlfriend on the West End. She talks to him.
........................................
"~I've got a room up Highbury way. I told them I was on the stage. It's a long way to have to come down to the West End every day, but it's a rare job to find anyone who'll let to ladies at all."
[Of Human Bondage by Somerset Maugham]
I'd like to know if "to have to come down" modifies "a long way."
Thank you in advance for your help.