Take him at his word. It is time for you to try to heal yourself, not try to make a wobbly relationship work. There are a number of unhealed features here for you that I believe warrant therapy. Your first marriage is a good example of something that should be explored and understood. The forces that brought you into a marriage, then changed your mind within a week are important to understand. The impulsive nature of the choice and decision is something worthy of understanding. You are at a point now where you don’t want to make another impulsive decision concerning your feelings. Rather than act on the relationship, I think your time would be better spent trying to understand your motivation behind it.
My strong encouragement is for you to seek therapy to understand your needs and motivations better. The find help tab at the top can help you locate someone in your area.