You are right to be concerned. It looks to me like your brother is sending up flares for help. He knows he needs it. He talks about being “gotten to” and “resisting” impulses he has. So far, he has managed but I think he is asking for help as clearly as if he was shouting it.
Your mother’s denial of the problem isn’t unusual. Many parents can’t manage the idea that something may be seriously wrong with one of their kids. So they justify and make excuses and rationalize behaviors they really shouldn’t. This isn’t a matter of “not defending you.” Her dismissal of your concerns is a reflection of her struggle to come to terms with the problem. Getting mad at her won’t help. Instead, have some compassion for her struggle and find a way to calmly talk to her adult to adult.
Talk to your mom about the importance of getting your brother help before he is in a situation where he can’t “resist” his feelings — not after he’s done something that gets him into serious trouble. The way your mother can love him best is to make sure that he gets therapy now.
If your brother is able to listen to you, do continue to try to talk to him. Instead of lecturing him, invite him to talk to you about what is driving the behavior and what he thinks he needs in order to have more self-control. Listen sympathetically so he knows he isn’t alone. Offer to help him get treatment. It may be a relief to him if you offer to go with him to talk to your mom about getting him help. Do some homework. Find out what therapists in your area work with adolescents with sexually problematic behavior so you can give your mother some direction.
Meanwhile, you are wise to keep your door locked. That protects you both.
I wish you well.