No, you can't omit "speaking" in that expression. You can of course omit that expression or substitute another adverb: The book is not a novel but a series of short stories. Actually, the book is not a novel but a series of short stories.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.