I’m so, so sorry you find yourself in the middle like this. It sounds to me like your mother is in some kind of emotional trouble. It’s not just that she’s having an affair. She’s stopped caring for children who are too young to be without a mother. You say this isn’t like her.
I think you need to tell your mother what you’ve seen. As hard as it will be, don’t accuse her or blame her. That will just make her defensive and angry. Instead, tell her how worried you are about her and how scared you are for the family. If she gets angry, tell her that you’re the one who has the right to be angry but you’re just mostly scared; that you need your mother; that you and the other kids need their family. I understand that this is a lot to ask of you. But you’re asking even more of yourself if you decide you and your sister should keep such a difficult secret.You love your dad as well as your mom.
Ask your mom to get you and your sister out of the middle. Suggest she get a counselor to help her sort out what to do. Tell her she needs to be the one to talk to your dad. It’s not fair to leave you feeling like you are betraying him by keeping quiet.
If all this gets to be too much, seek out a counselor at school or talk to your minister or rabbi if you have one. Professionals like these honor confidentiality. They can give you important support during this difficult time.
I wish you well.