a negative association can easily imprint on an animal, but you can overcome it. At risk of spoiling him for a few days, I would suggest that you sit with him on the floor after letting him out, talk very quietly and soothingly to him, and have a treat ready in your hand. Just open your hand, let him smell it, and even take it without reaching for him. Then if he comes back, take another treat and try holding it loosely in a closed hand and see if he will fish it out and then let you pet him. All the while keep talking to him gently. You are trying to reestablish an association with you that you also give treats and pets, not just harm, so he can’t ever be sure what’s going to happen but you probably can be trusted. You just don’t want to follow up the treat with a negative experience for him.
A bunny is more relaxed if you come at him with an open hand, palm up, from below and slightly to the side to start to pet his ears and forehead. They distrust a hand closed from above, and definitely a hand spread out from above looks like a claw to them, so try to avoid that gesture. He’ll forgive you soon.