If something is "fine by" you (also "fine with"), it means that you welcome it or approve of it. Harry welcomed an excuse to avoid being in Aunt Marge's company. In the next sentence, Harry tries to take as much time as possible moving the trunk, in order to avoid having to spend time with Aunt Marge.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.