Hi Viv,
I can understand your frustration, but please don’t give up. Most buns are not snugglebugs, but that doesn’t mean they don’t care about their humans.
Beny is still young, and many buns can take months or even years to develop deep trust with their parents (us).
Is Beny neutered? Unaltered buns can sometimes (but not always) be more manic.
Is Beny free range? Because you mention holes in walls… you could try bunny proofing your place by placing cardboard or something against the walls where he likes to chew and making him diggy boxes or chew distractions out of cardboard, phonebooks, etc.
One thing that really helps with bunny /hooman bonding is food and consistent hand feeding of even regular food like pellets and greens – it doesn’t have to be just treats.
Do you have or work near other pets or any smells you may not perceive but that might alarm Beny?
Let us know how it goes and remember that all bunny parents are on bunny time and bunny terms – so we sometimes have to just be patient for ages until the bun decides we’re ok and begins to trust us (they can sense frustration or stress as keenely as human babies).