"a lot of time" is not actually wrong, but "a long time" is more common and natural in this situation.
"go there" is unclear here. It does not clearly refer to journey time (which I am assuming is what you mean). It sounds like an uncertain blend of journey time and time until a decision to go is made or an opportunity to go arises.