Although you would not normally use "walk down" about an uphill climb, quite a large part of "down" here is idiomatic rather than literal, so having decided to use it, it feels odd to then change it on the basis of terrain. You could walk down/along the street and then up a/the hill.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
onizowalking down'down' and 'up' aren't often literally changes in elevation when used with 'walk' and/or 'street'.