You say that what the boyfriend says is true. If that’s the case, the problem isn’t the boyfriend. The problem is that you and your mother don’t have an adult relationship. Rather than blame the boyfriend, it would probably be more productive to look hard at yourself and take responsibility for your part in the fights — and fix them.
If you can’t do that, it’s time for you to move out and make your own life with your son.
Bottom line; It’s not healthy for your son to be watching the grown-ups fight all the time. It’s not healthy for the grown-ups to be fighting.
If you can’t work this out among yourselves, consider seeing a mediator or therapist to help you.
I wish you well.