I don't think you can say it's not grammatical, but "It is of no importance to me" or "It is of little importance to me" or "It's not at all important to me" all sound more natural.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.