What is his body language like when he hops away? Are his ears down, and does he hop away quickly? Or are his ears up (maybe turning but not pressed against his neck/shoulders/body)? Does he hop away a long distance, or only a foot or two?
If it’s the latter, I don’t think Ace is afraid of you. I think he just isn’t in the mood for cuddles!
It may be that Ace is just really interested in the outside world. He might feel the need to keep an eye on the garden/backyard/lawn/wherever it is you let him hop around in the morning, to make sure there aren’t any predators around.
The best way to get him used to you petting him is time and food. I recommend bringing out some treats for him and feeding him by hand (like yogurt treats if he likes those, or some celery, spinach, whatever greens he likes). You could also try just sitting with him, so that he gets used to your presence.
I hope this helps!