Your reasoning and explanation are correct. Let is indeed the reason. The plain/bare infinitive is used with let and also with active forms of make: Let's go !
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.