Usually I suggest that people trust their own instincts. But in a case like yours, instincts have been so battered that it is a real question whether they are trustworthy. You have been betrayed and hurt by many men. You have also betrayed and hurt others. So I think your gut is telling you that you have issues with trust, not necessarily that you have a problem with this particular man.
Deep inside you know that, however much you want to avoid it, it’s long past time for you to do some serious therapeutic work. You need to work through the pain of past relationships so they don’t color your current marriage. You need to forgive yourself for doing to others what hurt you so much. And you need to develop ways to work through the rough spots of a relationship so that the relationship can grow from them.
You would probably benefit from a combination of individual and couples therapy. I hope you will follow through on this. Much of your life has been painful. You deserve to finally relax into a loving relationship with the man you love. A few months of therapy can help you build your self-esteem so you feel more lovable and can help you and your husband build more trust in your marriage.
I wish you well.