"This dog, whose name I don't know, belongs to James." -- OK.
"This dog, which name I don't know, belongs to James." -- Wrong.
"This dog, the name of which I don't know, belongs to James." -- OK. Compared to "whose", this version tends to distance the speaker from the dog (figuratively, not physically).