It isn't a negative phrase, but it certainly carries negative connotations. Your suggestion as to its meaning is correct. There is an implication that the speaker feels it might be a waste of time.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.