This is a difficult question because there may be legal implications. For those, you may want to contact a lawyer as to the rights each of you have as parents.
But the psychological side of this would involve having a therapist become involved with the two of you — and the parenting that is happening now and in the future. I would highly recommend you both sit down with a couple’s therapist to have the concerns and arrangements for these concerns reviewed by a professional. This allows the primary concern you have about him being alone with his daughter to be discussed in front of a professional who is in the position of helping to make recommendations.
My point is that this is going to be an on-going concern. It isn’t about you having a date learn of this or the unusual nature of the arrangement, but that this concern will be ongoing as your daughter matures.
The presence of a therapist in both of your lives seems like a very safe, respectful, and legitimate way to go now that you are beginning to date. The Find Help tab at the top of the page will help you find someone in your area.
Wishing you patience and peace,
Proof Positive Blog @ PsychCentral