I’m sorry this situation is giving you so much pain. But I really don’t think it’s his past encounter with the girl at work that is the problem. You two just moved in together. That probably means that you are making a greater commitment to each other. Milestones like that tend to trigger any anxieties people have about their relationship.
It seems that you are with a man who has been much more adventuresome and perhaps less morally constrained than you have been. It could be that on some level you worry about whether you are enough for him or whether the differences in the way you have handled relationships is something to be concerned about. It would be understandable if you find it less stressful to be obsessed with the other woman than to deal with your doubts about whether you have what it takes to be this guy’s one and only. Another possibility is that you are sensing his anxiety about finally settling down. Such worries are normal and even predictable when people with such different histories come together.
Your boyfriend is right to be irritated that you keep bringing up that particular piece of his past. He is wrong to be mad at you for voicing your concerns. You are right to have concerns. You are wrong to be perseverating about your co-worker. What you and your boyfriend do need to be talking about is your mutual anxieties about the difference between your pasts. Can he reassure you that he is truly ready to give up his wandering ways? Is your self-esteem high enough that you can truly believe that you are the best thing that ever happened to him?
I hope you can have a loving and honest conversation about the real issues instead of getting sidetracked and going over his brief relationship with someone you know. You need to be talking about what moving in together means about your commitment to each other.
I wish you well.