Please slow down. What you are describing should be a red flag. Either your boyfriend doesn’t really want you to meet his children yet or the court has directed that he may not take the children on his own. Either situation should make you cautious. This is a man who left a woman with one small baby and another on the way. Of course the ex is trying to recover her family if she can. The two extended families are doing what they can to encourage a reconciliation, to know their grandchildren and create some stability for everyone.
As far as I’m concerned, the father should be going into counseling with the mother, not living with you. He helped make these babies. He has responsibility to do everything he can to make a home for them and care for them. However sweet he may be to you, he has lots of unfinished business with the mother of his children, the children, and his extended family. I worry that the relationship with you is a wonderful distraction for him from what he really ought to be doing.
You are only 22. You have plenty of time to find a Mr. Right. This one looks very Mr. Wrong to me. I think you should move out. If this guy goes into counseling, both with his ex and by himself, settles his court issues, has a clear and responsible relationship with his children and extended family and then still wants to be with you, maybe (only maybe), he is marriage material for you. If he doesn’t do all that, there is no guarantee that he won’t repeat the behavior and leave you and any children you might have together. Please take care of yourself and slow way, way down.
I wish you well.