When people say "should have" they usually contract it to "should've" -- which sounds like "should of." As a result, it's often incorrectly written as "should of."
There is no proper use of "should of [partciple]." "Of" does not functin as a helping verb. It's simply a result of people speaking "should've" and then writing it down incorrectly as "should of."