Good question! The correct is, "Is Ms. Bishop's Sunday school class going to sing "O Little Town of Bethlehem" in the pageant?"
Big items like books and albums are italicized. Smaller items - a book's poems and an album's songs - are placed inside quotations. In both cases, all words are capitalized except for smaller words, i.e. prepositions and articles.