Passive voice or adjective?

How can you tell the difference between a participle used as an adjective or as a passive voice clause?

For example:
"if they were phrased in the past tense, 'interested to' would be appropriate"

were + phrased = passive simple past or were = past simple and phrased = participle adjective. The same for the sentences below:

"I was interested by the man"
"I was interested in swimming"

Thank you!