He may not have had a physical relationship with her but he was cheating on you. Regardless of the other woman’s circumstances, he was directing emotional and mental energy to her that rightfully belongs in your relationship. We’re not talking about a momentary lapse here. We’re talking about three years of this nonsense. You even had to fight with him to change his phone number? That’s not a good sign.
My guess is he was getting off on getting away with it. He had lots of time in the last three years to break it off. It took being caught and threatened to put an end to it.
By the way, he’s just plain wrong when he says that “any other man will do the same.” Lots of men are loyal and faithful to their women. He apparently isn’t one of them.
I can’t tell you what to do. I can only suggest that you be very careful. Until he genuinely wins back your trust, your relationship is on very shaky ground.
I wish you well.