In spoken and informal English you would get away with it. I wouldn't write that in a formal work though. If an adverb is required, then an adverb it must be.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.