"of" in the "let go of me"

Hi,

I've found the phrase, "let go of me" from a fiction book.
Having searched all the possible meaning to understand the role of the "of" here, I have found a possible role of the "of".
The "of" here introduces the object of an action.
Which means "let go of me" can be paraphrased by "let me go" since "me" in the former, is object.
So they are only different syntatically, but the same semantically.
I hope I am correct for this one.

Could anyone please confirm that I understood correctly.

Thanks in advance.