You've got a good point there. The fact is that what you're looking for, have in the passive voice, doesn't happen very often except for expressions like a good time was had by all. It's just not used in the passive voice and that's it. No special reason for it.
You wouldn't say en bil hars av mig in Swedish either, would you?