"Very nice of you" means "That thing, which was done by you, was very nice".*
"Very thoughtful of him" means "That thing, which was done by him, was very thoughtful".
And so on.
In full, you need a subject at the start of the sentence (e.g. "That was very nice of you") but often in conversation the subject is dropped and implied by context.