As you may know, "little" emphasises smallness or insignificance, whereas "a little" emphasises the existence of something, albeit not in large quantity. There is no grammatical reason why "a little" shouldn't be possible in your sentence -- with different meaning, of course. " is a familiar phrase, almost a set phrase.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.