The apostrophe is required, but the sentence is not natural. Yes, behaving is a gerund.
Here are a couple of suggestions. The students’ good behavior greatly pleased the teacher. The teacher was greatly pleased [by / with] the students’ good behavior.