Only the second ('this seems not quite right') is correct, meaning that 'this is a little bit wrong'. If you want to say that 'this is very wrong', then you should frame it as 'this seems quite wrong'.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.