To me, "Let go of my hand" sounds more natural than "let my hand go." Maybe it's because when you keep "let go" together you get their meaning right at the beginning of the sentence. (Although I don't have a preference between "let me go" and "let go of me" .)