Your nephew is very lucky to have you as an aunt. Your statement to your brother-in-law was right on target. Our job as parents and adults is to love and protect children – regardless of how we were parented.
His parents don’t seem to get it that this little boy is showing every sign of being quite troubled. From what you report, it’s no wonder. Your sister and her husband are acting as if they wish that she didn’t have a prior relationship. Your nephew is evidence that she did. Your nephew isn’t stupid. He knows he is unloved and unwanted. He senses his mother’s rejection and clings to her. That only makes her reject him some more which makes him even more clingy — which makes her want to push him even further away.
In addition, I worry that this little boy is being abused. 5-year-olds generally don’t know about touching another person sexually. Often (not always, but often), it means that someone has touched them that way. Another possibility is that he quite accidentally discovered that touching his sister or himself in certain ways finally gets him the attention he craves.
As difficult as it may be, I think you should be contacting children’s protective services. Leaving him for 3 days at a time is abandonment. I’m not sure what you mean by “discipline” but if it is harsh, you may be skirting around the fact that it’s abuse. His sexual acting out may indicate something even more serious going on. Whatever the case, this little boy needs an evaluation and help.
I realize that calling in authorities can be terribly disruptive to relationships in the family. But if he is being abused, it needs to stop. I don’t have the whole story, of course. If you are uncertain about whether you should be taking action, it might be helpful for you to first talk to your nephew’s pediatrician or to make an appointment with a therapist to seek some advice and support.
I wish you well.