My bunny Joshua use to do that because he was handled poorly and not very well socialized. We spent a lot of time sitting on the floor with him and just waiting for him to come to us. Once he realized that we weren't going to hurt him, he started coming over more, and we'd hold vegetables to reward him with when he came over. A big thing was just to not act scared and let him know that we weren't going to push his boundaries. Right now he's much better even though he still gets worked up about food.
If you're scared that he might bite you wear boots and long sleeves. Don't show that you're scared because he'll sense that and think it's okay to 'beat up on you'. If he does bite you, thump your foot, and turn your back, that's rabbit talk for 'that's scary and you're beneath my notice'!
You may also want to check out: http://language.rabbitspeak.com/
The articles are very well written and will give you some big clues to what he's saying.