Only to have them waste my time

1) I went to their office only to waste my time.

2) I went to their office to waste my time.

3) I went to their office only to have them waste my time.

4) I went to their office to have them waste my time.

Which of these sentences can be used if it was not my intention to waste my time?

In other words, if the fact that my time was wasted was an unwanted consequence of my going there?

Gratefully,

Navi.