Clearly, it was not insignificant. These events are still having a negative impact on your life. You don’t trust men and aren’t open to a loving relationship. You deserve better.
Let’s take a look at what happened: Initially you and your brother were just kids. Adults often look back on things they did when they were young and wonder what on earth they were thinking! It does make sense that you enjoyed the sexual exploration at first. The body responds even when the mind is immature. But very quickly you realized you had crossed a line so you told your brother it couldn’t happen again. This is not at all an unusual story.
What is unusual is that your brother kept it up and even took advantage of you while you were asleep. As I read your story, I had to ask myself what was going on that made it difficult for you to tell him to stop and mean it. Apparently, all you felt you could do was try your best to avoid the vulnerability of sleeping in the same room. I don’t have enough information to help you understand that. A therapist who can hear your whole story would be able to help you figure it out and forgive yourself for that.
Sadly, your early experience with sexuality paired enjoying being stimulated with shame. At 27, these feelings are still with you. I doubt that you can move on without some help. I strongly urge you to seek out a counselor to help you come to terms with your feelings and to help you break the silence with your brother. It would be terrible if you were to go through life deprived of the pleasures of a trusting relationship and intimacy with a loving partner because of something that happened when you were a teen.
I wish you well.