I think your boyfriend does love you but doesn’t know how to cope with the activity and complications that come with a large extended family. On top of that, the closeness of your family is a painful reminder of the lack of it in his own family. My guess is that he may question himself as a father since his kids can’t seem to get along (and there are only 3 of them!) compared to your big gang. But his strategy of leaving when it gets too much for him isn’t working as a solution since he ends up angry and you end up feeling abandoned.
It’s way past time for a clear talk about the fact that marrying each other means marrying into each other’s family. You thoroughly enjoy your family and you’re not going to say “no” if someone needs you. He may not enjoy his kids as much but he still is connected to them. If you two want to be with each other, you need to work together to find ways to reduce the stress for him. This isn’t about a contest for your affection. When people get together as adults, they have pasts and connections. Part of becoming a successful couple is learning how to incorporate both families (yes, both — his kids too) into your new life together. I suggest you see a couples counselor to help you good people solve this practical problem.
I wish you well.