This is ok if the same group of people go swimming and horse riding. Otherwise, you need either
Some go swimming and others go horse riding.
or
Some go swimming and some go horse riding.
Some people might prefer 'at the weekend' but 'at weekends' is fine too.
In AE, you would use 'on weekends'.