I’m sure this is very worrisome and painful for you. You are right to be concerned. Your doctor was right: It is normal for a 3-year-old to go through a stage of curiosity about bodies. But it usually subsides way before age 7.
Please make an appointment with a child therapist. I don’t have enough information to go on. A counselor can help you get to the cause of this behavior and will give you guidance about how to handle it.
The most benign possibility is that you have become so worried about his behavior that your son is simply trying to get extra attention and concern from you. He’s learned that all he has to do is tug his brother’s pants and he gets your full-on attention. For some reason, he’s decided he can’t get the same involvement by doing positive things. In that case, you need to learn other ways to avoid reinforcing his inappropriate interests and at the same time give him positive ways to be engaged with you.
At the other end of the spectrum of possible causes for this behavior is that your son has been abused sexually by someone and is acting out what he is too scared to talk about. This is not at all uncommon in kids. In that case, you need to protect him and get him treatment.
Or his behavior may be about something else entirely. That’s why you need to see a therapist. You want to respond to the real cause to get him past whatever it is that is troubling him.
This has already gone on far too long. Please see a counselor and get the help you need to steer him to more socially appropriate behavior before he gets any older. If he’s been hurt by someone, he needs treatment. If it’s attention-getting behavior, it’s likely he will be bullied and isolated by his peers. Of equal concern is that he may become very confused about sex and sexuality as he gets older.
I wish you well.