Unwanted touch is unwanted touch. This man violated your sense of personal space. As a young child, you couldn’t understand sexuality but you could sense that something was wrong. It’s positive that your parents did listen to you and that the family gradually stopped visiting.
As for the masturbating: Often young children do what is called a repetition compulsion. They repeatedly play about or imagine something bad that happened to them as a way to try to make sense of it. Unfortunately, the intrusion by this man got connected for you with sexual feelings.
And also unfortunately, the more you masturbate to fantasies of this man, the more you will probably do it. Orgasm is a powerful feeling of pleasure and release. In your case, it is reinforcing the masturbatory behavior. You need to find another fantasy that arouses you and substitute it for the ones you’ve been using.
This is important. If you don’t separate sex from an abusive fantasy, it will be difficult to have an intimate, loving, physical relationship with someone you love.
I wish you well.