"To hand it to you" is ok is expressing a direct transfer of the parcel from one person to another, bu in the context of a failed delivery, I would expect the sentence to be "The postman/courier tried to hand it over to you on Friday".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.