I don't think it is rude either. At the destination wedding I went to early this year, the bride and groom had a rehearsal dinner on Friday (people arrived either Thursday or Friday), afternoon ceremony and cocktail party on Saturday, dinner at one of the restaurants for a reception Saturday night. They also spread the word that there was an optional cruise on Sunday afternoon, but we decided not to go.
Otherwise, people kind of split up and formed their own groups for activites (snorkeling, eating, going on excursions). There were ~35 people, and this was at an all inclusive, so it was easier this way. It was nice being able to run into people you knew and could talk to on the beach or at the buffet. It might be harder with people spread out at different hotels or villas.