This is the most usual way of saying: "this is the man by whom you were followed" 'you were followed by a man/this is the man) So yes, "by" is part of the subordinate (relative) clause.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.