Of course you should say something. Holding in these feelings will not do you — or your relationship — any good. He needs to hear this and you need to say it.
The important point is to do this without blaming him. Stay focused on your reaction and what it is that upsets you — not on blaming him. Use ” I” statements, not “you” statements, so that you keep the information you are sharing with him about your thoughts and feelings. This will allow him to hear you without having to be defensive.
Finally, I would recommend saying something sooner rather than later. I believe this will allow you to develop a more honest and trusting relationship.