It’s unfortunate you didn’t address it at the time. But it’s never too late to deal with a situation like this. I think you are correct to arrange to talk to him — for all the reasons you stated. He needs to understand about consent. He needs to understand that your lack of reaction doesn’t mean that you were giving consent. He needs to understand that being underage doesn’t mean he isn’t responsible for his choices.
I am concerned that you were afraid of his reaction. I hope you can tell your husband what happened and can ask for his support. Ask him to accompany you in confronting the boy. The boy needs to know that men don’t think this behavior is acceptable either. Alternatively, you can ask your husband to be in the next room and “on call” if things get out of hand.
If there are others in the family who are vulnerable (like young cousins, for example), then it’s important to do what you can to protect them. That may mean telling the boy’s parents about what happened even though “all hell would break loose”. The safety of younger children is more important than keeping the boy’s approach a secret. If you are certain that he is limiting this behavior to you, then you can let him know that you are willing to deal with this privately for now. But make sure he understands that if you hear of him ever doing something like that to anyone else in the family or to anyone who is underage, you will immediately tell his parents.
This is a tough situation, and I’m sorry you have to deal with it. But it’s our job as the grown up to give kids a clear message about right and wrong. You know that. That’s why the whole thing is bothering you. As difficult as confronting the boy will be, I think it will be harder for you to keep such a troubling secret.
I wish you well.