It’s important that you talk to your parents. They don’t have to know you were on your phone. It’s enough that you “woke up” to your brother being in your room and apparently masturbating. This is a form of sexual assault and it has to stop! You need to feel safe. Your brother has a serious problem that needs to be addressed immediately.
Setting up taping seems difficult and maybe risky. I worry what your brother would do in retaliation if you resorted to that. You could suggest to your parents that you could do taping if they don’t believe you and see what they think. (The suggestion by itself may stop your brother from coming into your room.) That may solve the immediate problem, but it doesn’t address your brother’s lack of understanding about appropriate boundaries or his apparent inability to control his impulses.
I do think you need a lock on your door. It’s normal and appropriate for a 16-year-old girl to have privacy and security. A lock would at least take care of keeping you safe.
I hope your parents will take you seriously. You deserve to feel safe in your home. And your brother needs help. If your parents won’t take action, do consider talking to another trusted adult who might help you get through to them. Adults will sometimes pay more attention to another adult than to their child in situations like this.
P.S.: Stay off your phone in the middle of the night! Research is showing that sleep deprivation among teens is now an epidemic. You may not realize it, but being sleep deprived can cause all kinds of psychological trouble. Kids who get less than 8 hours of sleep a night often they develop anxiety and/or depression. They often have trouble with concentration. Grades go down. Driving accidents go up. Bottom line: No one can function at their best when they don’t get enough sleep. So put your phone in another room at bedtime and give your brain and body the rest they need so you can be healthy and successful.
I wish you well.