I’m sure this is very stressful. It looks to me like you are both struggling. It is natural and normal to have sexual feelings and to want to be intimate with someone you love. But when acting on those feelings is against everything a person has been taught about what is good, right, and proper, it often creates an internal struggle that is very painful. If you are physically intimate, you are behaving in a way that is against the family and cultural values you’ve been taught. If you don’t allow yourself to be intimate at least to some extent, you feel like you are missing something important in a relationship.
One way to avoid the dilemma is to avoid each other. It looks to me like he is solving the problem that way. Your boyfriend is taking distance. You, on the other hand, tried to find a way to enjoy some physical intimacy by communicating “rules” that work for you. The problem is that you didn’t talk together about how to be physically close at a level that you both find acceptable.
I’m guessing that you haven’t known each other long enough to develop the trust necessary to talk as frankly as you need to about the issue. It can be scary to be that vulnerable. A talk about how you are going to support each other while talking about things you were taught are forbidden is the place to start. If you do that successfully, you can then talk about the tension between your upbringing and your physical desires and specifically how you are going to manage it.
I wish you well.