I do think it is unhealthy for you to hang on. If you don’t find a way to resolve this with him — either by making it better or separating — do you really want to have a life where all you are doing is hanging on?
He has been unfaithful over and over. I think that as uncomfortable as it will be to confront the issue directly — I don’t think living with your anxiety is the way to go.
I recommend a couples therapist to do two things: first, evaluate your relationship to see if there’s enough there to work on. Secondly, if there is then work together to help improve your relationship. If not, have the couple’s therapist help you separate.