"gone to the dogs" is a fixed expression. You can't change it in any way,* irrespective of what general grammatical rules might tell you. * Edit: Well, you can change the tense, as in "going to the dogs".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.