I don't see that there is any difference in meaning, but I think the choice of preposition depends on the object.
I am annoyed with her. I am annoyed by her. I am annoyed at her.
All of those seem fine if the object is a person, and the sentence stops there. But what if the annoyance is caused by a specfic action or other object?