It is a common phrasal preposition (or adverb + preposition) that some grammarians eschew because it seems redundant – so stylists often omit 'of'. You cannot write 'off've', though, I think, as a transcription, since ''ve' has been preempted by 'have'.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.