With his history, it seems like it will take a while for him to fully trust people. Hopefully, with time and patience, he will come to accept and like the attention. Most rabbits do hate being picked up and they can be hard to handle when they are squirming and flailing around, and can also get injured, so you do have to handle them very carefully, maybe wrapping them in a towel or blanket to make them more still. If you did ever have to take him to the vet, you would want to have a carrier right there at hand. You could try to lead him into it with a treat if it is side opening, or just quickly pick him up and get him in it before he realizes what is happening.
As far as playdates, rabbits should be carefully bonded through a process. Bonding him would mean that the other rabbit would live with him on a permanent basis. Occasional casual meetings with other rabbits would not work. It would cause confusion, and there would likely be fighting resulting in injuries if proper bonding procedures weren’t followed.
Is he neutered? Before any bonding can take place, both rabbits should be spayed or neutered and recovered from their surgeries.
Here are a couple of good articles on bonding:
https://binkybunny.com/BUNNYINFO/tabid/53/CategoryID/9/PID/940/Default.aspx
http://rabbit.org/faq-bonding-multiple-rabbits/