I’m sorry to have to tell you but this relationship is not “all good.” Your boyfriend is making unreasonable demands for constant access. You are at his command. When you legitimately couldn’t respond to him, he punished you by cutting off contact. He is still punishing you. This is what is called emotional abuse.
The nicest thing he has done for you is show you how unreasonable and controlling he is. You deserve far better treatment from someone who says he loves you. However good the good times are, they aren’t worth having to put up with this kind of behavior. From my point of view, losing him would be like losing a stone in your shoe. You will miss him for awhile, just because you are used to the pain. But eventually it will be a relief.
Reclaim your self-respect. Stop apologizing. You have nothing to apologize for. Leave this guy and make yourself available for someone who knows how to cherish and care for someone he loves.
I wish you well.