You are correct - detour would not be used passively. So: "The only time (not the only one time) that I need to (not do need to) take the bus, it has to make a detour". A route is not detoured - that does not make sense.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.