While I appreciate that these memories and feelings are difficult, I believe you are doing the right thing by asking this question about your past. The discomfort you feel, the memory of the journal, and the history of fear and intimacy problems with men would make sense. I admire your bravery.
I would suggest that you work in individual therapy with a seasoned therapist familiar with early child abuse. Someone with experience will be able to help you sort through these symptoms and memories, and help you find some answers and therapeutic interventions. Reaching out her was a good start — now I would follow it up with individual therapy.