No, you are using the verb form of "speak" and you need a gerund or an infinitive to act as the subject of the sentence. So, either To speak to an audience is always stressful. OR Speaking to an audience is always stressful.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.