If you’re worried about him, it may be a good excuse to go for a general check up at the vet, maybe get his nails trimmed while you’re there, make sure his teeth are fine and just tell her what you heard so she’ll examine him for any irregularities. It sounds like he’s OK, but we tend to wait with bunnies, so if you get him seen you won’t need another visit for a while, hopefully.
It is true that they do not like to be picked up or held, and they can damage themselves (severely at times) so you must let him go right away when he struggles. Just don’t let him fall from a height. He will rarely sit in your lap, or stay there, but you can let him play on the couch with you on it and many bunnies will settle down next to you for the companionship, even if not totally on you. He needs to feel safe around you, not afraid he will be grabbed hard. (I know it’s kind of hard to let go since they are so soft and cuddly.)