Bonding with a bunny and earning his trust takes time, patience, and willpower. One of the first steps is to resist the urge to reach for him and pet him. You’re doing fine sitting on the floor with your ds, but it sounds like he isn’t sure he can trust you yet. When you’re on the floor, try talking softly to him (even whisper), but don’t reach out and don’t try to pet him. He will eventually become curious about you and come over to check you out. Let him get used to being close to you without being touched. (I lay on my stomach on the floor and my bunnies hop around on my back). Move on to offering healthy treats. Once he’s comfortable with that, you can try a little head rub followed by the treat. He may never be a lap bunny (most bunnies aren’t), but if you take baby steps with him, he will come to trust you. With bunnies, everything must be on their terms or all bets are off!

You’re saying that you need to pick him up to put him in his carrier – is there any way you can encourage him to go in by himself? Put treats inside, for example? When we first brought Puck home, we left his carrier in the cage as a hide for him so that he wouldn’t find it strange or intimidating when we needed to put him in to take him to the vets. We’ve since got him a new hide, but I still sometimes put his carrier in there so that it doesn’t become unfamiliar and potentially frightening for him.
I second what JackRabbit has said about giving him time and refraining from trying to touch him right now – he needs to know that if he doesn’t want to be touched that that’s ok and you won’t force it on him; remember how much bigger than him you are and how frightening it might be if someone proportionally larger than you wanted to pick you up and cuddle you all the time while you’re still not sure if they’re trying to pick you up so they can eat you! If you’re anxious to make progress, he can probably sense that, even if he doesn’t understand what you’re anxious about and it may make him more skittish.
With Puck I spent the first few days sitting in a chair next to but outside his pen and just read a book, chatted with him when he came to the side of the cage or pen, but mostly left him to it. After that I sat in the pen with a book or sat in the pen while I watched tv – I was there for him to explore but I was otherwise occupied, so I wasn’t staring at him or anxious to “do something” that felt like progress.
Obviously I’m not an expert, but that’s what worked for me.