I’m impressed that you want to solve this problem instead of merely bailing. It shows a lot of maturity for a 16-year-old. In spite of the ways you are feeling mistreated, there must be some heart connection with your mom.
I would have to know a lot more to really help but perhaps I can offer a few thoughts: It sounds to me like your mom is barely holding it together. It’s not normal to hit and curse at a son. It’s as if she needs you to not need her in order to cope. When you show your disapproval, it may remind her of all the ways she disapproves of herself. It’s not unusual for a single mom with a baby and a teenager to be overwhelmed. Managing a household, a small child, perhaps a job, and perhaps financial stress puts many single parents on edge. It doesn’t take much to push someone in this situation over it. The closest person then gets to be the scapegoat.
In an ideal world, a 16-year-old shouldn’t have to think about taking care of his mom. But the world is seldom ideal and you’re asking for advice. My best suggestion is that you try to have a quiet talk with your mom when things are calm. Ask her if there are ways you can help relieve her stress. Tell her that you hope she doesn’t mean the things she says when she gets upset but even if she doesn’t, they still hurt. Encourage her to maybe think about either some family therapy or some individual therapy to help her manage all that she is juggling. Throughout the conversation, be your most mature self. Take a stance of concern, not accusations or anger.
I can’t guarantee it will work, of course. But it’s worth a shot. It seems to me that the worst thing that can happen is more of the same and the best thing would be that she would be appreciative.
If it doesn’t work out, then find constructive ways to be out of the house more. An after-school job, a sport, or involvement in the arts will give you a place to go where you can feel good about yourself. When you are home, be as polite and low-key as you can. Perhaps focus on your little brother and just give your mom a little break from child care. Hopefully things will begin to change for your mom so she can be less irritable and you can get closer.
I wish you well.