How frustrating it must be to feel simultaneously so connected and disconnected to your boyfriend. But your concerns are real – so let’s sort through the options.
The fact that he lied and was interested in staying in touch with the ex and not telling you about it tells us a lot about his character, and why you feel as you do. The phone episodes also make his comments and actions suspicious.
The bottom line is that you don’t trust him and for good reason. What couples often fail to realize is that if one of them has a concern, then there is a problem in the relationship. Your anxiety was spawned by his actions. That is real, not made up. He simply hasn’t earned your trust and this will not go away until it is addressed directly by the two of you as a real issue.
My strong encouragement is to have the difficult talk about this with him, and try to find ways of dealing with your concerns with him directly. When he says you can come and check on what he is doing you will have to decide if that is the route you want to take, and what he can do to help alleviate your anxiety. If this isn’t successful you may want to find a couples counselor to work with.