The clumsy thing is weird. Does he ever shake his ears or tilt his head? That can be a sign of mites (also known as ear canker), which can throw off the rabbit’s balance. He may have had them in the past, and developed a balance problem.
Try stroking him from his ears to his rump in one continuous stroke. It has worked for me to calm down aggressive rabbits. Rabbits remember. If his previous owner treated him badly or neglected him, he probably does not trust people in general.
Many rabbits bite if you invade their territory. My bun Panda, one of he sweetest bunnies ever, has bitten me through the skin many times because I was in her space. Does he bite when you feed him, give him water, and clean his hutch?
Climbing on you and chinning is a very good sign. He enjoys your company and trusts you enough to sit on you. Some rabbits just don’t like being handled at all, but are fine being your buddy if you just let him relax. It may be the angle your hand is approaching that throws him off. Let him sniff your hand before petting him, then try scratching his forehead and between his ears.

A lot of it sounds like hormonal behavior. I wonder if maybe he wasn’t neutered properly or it was incomplete. It can happen.

The yelling is concerning though, as it’s a behavior they only do in utter fear. To be honest, I wouldn’t suggest touching him or petting him at all. That can take a lot of trust for some bunnies. Especially as a rescue you don’t always know what the previous situation was like. He could’ve come from utter abuse and neglect. I’d stop trying to hand feed him as well because some bunnies are very food aggressive.

The best thing you can do is ignore him. I know it seems counter intuitive, but the best thing you can do to gain his trust is just doing your own thing, and stay on the floor with him at his level. Don’t pick him up, and don’t try to put him anywhere else. You go to him. Some bunnies never enjoy being pet, and as much as it sucks for us, it’s something you have to respect. My boy Asriel, I’ve had since he was 8 weeks old, and he’s almost 2. He still doesn’t not like being pet. You can give him some head rubs now and then, and he’ll hang out with us, but he doesn’t not want to be touched. Sometimes a bunny showing trust is just being there in the same room a few feet apart. If you stop trying to touch him or interact with him, the aggressive behaviors may slowly go away.

Something to keep in mind is that the human body is very large. While you know your arm coming towards him is simply an extension of your body, he may not readily know or believe that, and becomes defensive at this new “thing” approaching him. Floor time is important, especially with no interaction, so a rabbit can learn you wholly, I find.