A few years ago I could afford to bring my wife to this place for dinner, too

George worked in San Francisco. He wasn't married, and he usually had his lunch, and occasionally his dinner in small restaurants. One evening he decided to go to an expensive restaurant, and when he got in, he saw a large, heavy man who he had not seen for several years sitting by himself at a table. He thought for a few minutes and then remembered the man's name, so he went up to him and said politely, "Hello, Mr Grey. How's your business?"
"Oh, it's not good at all," the large man answered.
George looked at the expensive food and wine on Mr Grey's table and was surprised.
"It certainly doesn't look as if your business is bad," he said.
"Well," Mr Grey answered sadly, "I'm afraid you're wrong. A few years ago I could afford to bring my wife to this place for dinner, too."

Could you tell me what could the man mean by "A few years ago I could afford to bring my wife to this place for dinner, too"?