hey, both sentences are incorrect badly-smelling dog and badly smelling dog. Because badly is an advervb and adverbs qualify the verb. In this case you're qualyfing the noun dog, so the correct form is bad smelling dog.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.