Poor Benny! Do you know his story, where he came from etc? He may have come out of a really bad situation with groups of people.
Try inviting one person at a time for a while. Also, no loud (think loud for someone who can hear way better than you can) music or loud conversations, because your bunny can’t tell if your friends are being aggressive toward you or him if their voice is raised. You and your friend should sit down and gently play with him, then leave.
More repeated interactions like this should help. You can then add another friend, etc. I think if your friends are big on horseplay or boisterous conversations, Ben will probably never really love that part. But bunnies are social, and a happy bunny can be a really awesome thing for guests when they come over. They love it!
So sorry about your bite though
Is is healing ok? Clean it well and take it as a sign that Ben is really scared, either for you or for himself, and you need to reassure him. It’s important for your relationship!