If you're talking about one type of skill/ability (e.g. writing, reading, running), then use the singular. e.g., The students were grouped according to their singing ability. or The students were grouped according to their ability in writing.
If you're talking about several types of abilities, then use the plural. e.g, The students were grouped acc