"Of" does not imply that anything has ever happened. "By" implies that you have experience. You can be frightened of many things, but if you're being frightened by many things you may want to move to another neighbourhood.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.