Sometimes we love people because they’re not available. I think this is likely to be the situation in this case. There are several things about unavailability that can allow us to feel feelings while remaining relatively safe. We can have all of the feelings without all of the actual bother of the relationship.
Second, it is very common to be drawn to someone who is similar to a parent. Many of my clients who have unavailable partners have at least one distant parent. This is also something the relationship scientists study when they look at attachment. This might account for why somebody you see so sporadically could be eliciting such a strong attraction.
Typically, once there is some genuine intimacy and closeness in these unavailable relationships, it can’t be tolerated. My best guess is that this is what happened with him. As soon as the feelings got too intimate, the relationship lost value for him.
I would highly recommend a few brief individual sessions with a counselor to help you through this adjustment period, and to learn more about the strength of this attraction.