I appreciate how difficult speaking up for your needs can be — yet I deeply admire your courage and strength in doing so. Let me offer three possibilities. I am sure there are other options, but these give you some leeway in responding.
The first is a heartfelt letter that you would write and distribute to the group ahead of time explaining your love and decision not to come. This will allow you to maintain a connection, share your love, and yet not be expected. Secondly, you may also want to consider going, but staying very briefly. This strategy has worked when, for a myriad of reasons, a family member knew they could not stay too long in the presence of so many people. The work here is to be certain you have a place to go, be able to connect with the people you want to connect with, and then be certain to have a destination to go to afterward. Finally, a simple decline to the organizer of the event should do it — with regrets and not a lot of explanation.
The important thing regardless of what path you chose is to honor your feelings and find a way to make them known and act on them.