Yes, you should be upset. The young man is violating the privacy and boundaries of the women around him. Although I’m concerned about all the women, I’m especially concerned about how this behavior affects the 12 year old. How is she to feel safe in her home, if she can’t rely on privacy when she sleeps? Put a lock on her door until you take care of the problem.
I’m sure the young man knows his behavior is not okay. (That’s why he makes excuses.) I’m reasonably sure it’s part of why he does it. There is excitement in trying to get away with the forbidden. It’s also possible that, at 19, he hasn’t figured out how to leave home and may be unconsciously creating a situation where you will have to kick him out.
A lecture about privacy from you or your boyfriend isn’t likely to change things. Setting clear consequences might. Make it clear that at 19, he’s an adult. You are not obligated to provide him with a home. You are obligated to keep your household safe. He therefore has to leave. (Set a time limit.)
If he says he was just doing it to rile you and won’t do it again, trust your instincts. Decide if you trust him enough to let him live with you. If not, he should leave. If he says he can’t help it, tell him that he needs to get some therapy to learn to manage his impulses and he still has to leave until he does.
He will probably test whatever line you draw. Make a plan for when that happens and stick to it. He needs to get this resolved. Someone isn’t going to be as forgiving as you are. He could end up seriously hurt, in jail or both.
I wish you well.