This isn’t so much selfish as it is a struggle with genuine intimacy. You fell in love with one man that was a long-distance relationship, unavailable by geography. Then you fell in love with another man who was unavailable because of his marriage. You are likely not to be in love with each man as he is, but rather in love with his potential.
The man you are engaged to is now becoming more available. You have worked though the rough spots and now are on the threshold of becoming an intimate couple.
A struggle with intimacy means the more available a person becomes, the more difficult it will be for you to make the commitment. This is my best guess as to what has happened. If I am right about this, then swapping out one man for the other in the name of love is not likely to yield a true change in your capacity to tolerate intimacy.
The married man and your fiancé create the same dynamic inside of you. You want and cannot fully have either.
Do not worry about hurting feelings, being selfish, or trying to do the right thing. Concern yourself with coping with the internal feeling of wanting and not having. The source of your dilemma is most likely coming from a dynamic originating inside you that developed from unresolved issues.
I would make an appointment with a therapist. Our find help tab would be a resource for finding one in your area.
In the meantime I will encourage you to read the book Dance of Intimacy, a classic in the field that might shed more light on this process for you.