Neither is natural with that verb form. -- Here, the agent is not the feather, but the person who is tickling you with it. 'By' generally refers to the actual agent or 'doer'; 'with' refers to a medium or instrument.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.