If you want to describe a person moving backwards, then "back off" is OK. Others you might consider are: "taken aback," disheartened, or disappointed.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.