"affect" means simply - have an impact on. It can be negative or positive.
If you say that a negative thing e.g. drug trafficking, affects someone, the implication is that it affects them negatively. And vice versa. There need be no ambiguity.
If the thing is neutral, or the context otherwise demands greater clarity, you would need to modify "affect" to describe it positively