Does he have full run of your house or of a large area in your house? You may find it helpful to set him up in his own small area to begin with (such as in an exercise pen). Once he seems comfortable about this space, expand his area. When his area is big enough for you to fit in with him, try hanging out quietly. I once saw someone suggest (not sure if it was here or another site) reading a book or checking your phone while sitting in the bunny’s space.
My bunnies love their pellets and we were able, over the course of a month, to get them to jump on top of us (while we are lying on our backs) by placing pellets on our bellies. 🙂 Neither of our bunnies likes to be held and only Luna appreciates long pets.
There’s plenty of time to bond with your little guy, even if he never ends up being a cuddle bug. I think, given time and patience, he will eat out of your hand eventually (particularly once you find a favorite food for him). Is he old enough to get into vegetables and such? If so, see if he would like a baby carrot or some parsley (my buns love their parsley).