You didn’t mention how far apart in age you and your brother are. Many kids who are close to the same age explore sex with each other. They keep it up for a time because it feels good. Sometimes kids’ needs for physical affection aren’t filled. If you were getting very little from your parents, being sexual with your brother may have been especially important to the two of you as it was a way to be touched and to touch. If, however, you brother is older and was in a position of power or authority over you, then what you describe would be termed abusive. The report from your friend is disturbing and supports that possibility.
I think you need to sort this out with a therapist. You had legitimate needs as a kid. You have many questions now. Some work with a therapist will give you perspective and a way to put the shame and self-blame behind you. There is no need to carry the hurt and troubling feelings into your future.
I wish you well.