No one has the right to harass you. He has been keeping tabs on you for the last six years. Technically this behavior may fall under the definition of stalking.
I’m concerned that your reaction to this situation may not be appropriate. It’s important not to overreact but based on the tone of your letter, you seem more concerned about offending your ex than you are about your safety. I know you don’t want to jump to conclusions but it is a distinct possibility that he has been stalking you for a significant length of time. Don’t take this lightly.
Stalkers can be dangerous. It’s important to be realistic, to act accordingly and to protect yourself.
Your suspicions should be reported to the police. Filing a police report doesn’t mean he will be arrested. The police will investigate, review the evidence, advise you about your rights and determine legally how to proceed. When you file a police report, be sure to bring with you any evidence you may have gathered such as letters, e-mails etc. This evidence will be an important aspect of their investigation.
Understandably you want this to harassment to stop. That is your right. I hope you will take the necessary actions to investigate his behavior by reporting this situation to the police. Even though it’s a cliché, it’s fitting in your circumstance: it’s better to be safe than sorry.