"That is quite a different matter" is more idiomatic than "That is a quite different matter". In the sense you're using it, "matter" is countable ("a matter" = "a subject/thing/affair"), so an article is necessary. "matter" can also be uncountable, referring to physical substance (as in "everything is made of matter"), so "That is quite different matter" is actually grammatical, though rather an unusual thing to want to say.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.