I hear it most frequently, and see it in writing, to describe a person's character or past, e.g., "He is an unsavory character that I would prefer not to be around." Or, "He is rumored to have an unsavory past, but not much is known about it."
It can be applied to food or anything else, but I seldom hear it used any way other than exemplified in the sentence above.