I don’t think it’s social anxiety. And I don’t think you couldn’t fit in. I do think you need to do some work on your social skills. Being comfortable interacting with other people is a learned skill. It’s easier for some people than for others, it’s true. Some people are just born extroverts. But the rest of us have to learn how to initiate and maintain conversations; how to enter into a situation where other people seem to know each other; and how to be at ease in a crowd.
Instead of sitting all by yourself hoping for someone to notice, you could have joined a group and started acting interested in getting to know other people. Most people respond positively when someone starts showing interest by asking questions and listening attentively. Most people decide someone is interesting when that person is interested in them.
Read up on communication and social skills. Talk to your brother about how to connect with his friends. Practice with your own friends and, especially, with friends of friends you meet. Practice does matter. With practice, you too will be able to be comfortable in settings where you don’t yet know another person there. Remember: Strangers are often friends you just haven’t met yet.
I wish you well.