Yeah, I wouldn't want to omit 'of' here. As it is, your sentence looks OK to me. In fact, without the 'of', the sentence would look a little odd, incorrect perhaps.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.