There are many possibilities here that have to do with your past, your struggle to pay attention in the present, and the fidgeting. But I would not rule out social anxiety for accounting for at least part of these reactions. This blog by Johnna Medina, Ph.D. explains the conditions surrounding these reactions and offers suggestions for improvement. Most people who have social anxiety recognize that their anxiety is out of proportion to the situation and is unreasonable. Although more women struggle with social anxiety than men, men more often seek treatment.
Here are just a few of the conditions that go along with this condition that are elaborated on in the article by Dr. Medina:
• Exposure to the feared social situation almost invariably provokes anxiety, which may take the form of a situationally-bound or situationally-predisposed panic attack.
• The person recognizes that the fear is excessive or unreasonable.
• The feared social or performance situations are avoided or else are endured with intense anxiety or distress.
• The avoidance, anxious anticipation, or distress in the feared social or performance situation(s) interferes significantly with the person’s normal routine, occupational (academic) functioning, or social activities or relationships, or there is marked distress about having the phobia.
But, rather than guess, I would highly recommend an evaluation by a neuropsychologist or clinical psychologist. Many times, such evaluations can be illuminating to the therapeutic process. With a battery of tests, a neuro- or clinical psychologist can typically identify your strengths and weaknesses relative to one another In this way, you may be able to figure out a course of treatment, medicine that can be helpful.
Wishing you patience and peace,
Proof Positive Blog @ PsychCentral