I am new to bunnies but a dog trainer by profession so have a bit of experience dealing with behaviour.
With most animals food is your friend, it is a primary reinforcer for all animals because it is required for survival therefore it is usually high valued.
I would work on having some yummy treats handy (might be part of his daily greens or pellet allowance even) and every time you get up to move throw a small amount of food towards him so he begins to associate you moving around with food, at the moment he associates you moving around with something aversive/scary (being shooed), I also would avoid forcing him into doing anything, that is called flooding and when an animal is already feeling nervous or fearful it is likely to make them feel more so. Instead reward all social behaviour with food (again use his daily meals) and gentle petting/attention. He will slowly build confidence as he associates you with all the good things in life and you will gain his trust by not pushing his boundaries.
Hope this helps.