It can be viewed as a reduced clause (the suggestions are my own): (Let) Every man (act according) to his taste. Dudley Do-right (is coming) to the rescue! (Permit) To each his own (preference).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.