"in the library an awful lot" = in the library very often, and/or for long periods of time "Quite apart from X, Y" is presenting two reasons, X and Y, why Harry was annoyed. "apart" literally means that Y is separate or different from X. " is a set form of words.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.