" In meaning, the expression is similar to "should" and "ought to", and it's very common in conversation. I've heard some people omit 'had' in sentences like that. I assume but I'm not sure that it's a more informal way of saying the same thing (that would be your second sentence).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.