I would never tell someone what to do. It is up to you to decide. What I will suggest is this: Your boyfriend has been through a terrible experience as a parent. His heart has been broken by the distance put between him and his daughter. I do understand why he doesn’t want to risk it again.
But there are other ways to handle this besides depriving himself and you of a positive parenting experience. I’m surprised and sad that he and his ex didn’t get some family therapy during their conflicts. It would be better for the child — and for the adults — if the two of them learned to be supportive coparents even though they couldn’t be good partners for each other.
It isn’t too late. For the sake of their daughter’s long term mental health and for her ability to make a positive love relationship in her future, I hope they will consider seeing a therapist now.
If his ex won’t cooperate, he can still go himself. Or the two of you can work with a therapist to help him come to terms with his grief and regain his courage to bring a child into the world with you.
You will be in a better position to make your decision once you see what happens in therapy.
I wish you well.