A pair is always two of something. A set could (in theory) be any number greater than one, so a pair might constitute a set, but in practice one would normally say there were pairs of marbles, not sets, in your example. A set would suggest a group containing certain specific items (for instance, one red marble, one green marble, one blue marble, one white marble and one black marble), while a pair would only indicate that there were two marbles.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.