When a past participle is used in forming a finite verb structure, it will appear with the auxilliary "to have" (perfect tenses) or "to be" (passive structure). So you're safe in assuming that in "I'm starting to get worried," "Worried" is used as an adjective. Often the passive structures (to be worried) are still ambiguous, and may also be viewed as adjectives.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.