You are on a wonderful site to learn everything you could ever want to know about bunnies, so spend some time reading through posts and seeing what other people do and how their bunnies behave. Don’t give up on your bunny please. There aren’t enough good bunny owners, but if you put in the work to learn, you can be one!!
It takes lots of patience. I’ve had mine just over a year, and only now I feel like I know my rabbit very well.
Like the others have said, baby bunnies don’t mind to be held, but adolescent and adult rabbits do not. It scares them. Simply avoid picking up your bunny, unless it is absolutely necessary. Bunnies are borrowers, so they naturally like to dig. Make a digging box for him/her and fill it with shredded paper. If he/she nips at you, its because they don’t want to be held. And the growling is him/her being territorial. My bunny has 2 territories that I have to respect: her cage, and under my bed (she has claimed that as hers). So I don’t reach under my bed, and if I put food in her cage, it is when she is not in it. Above all, be patient! You will get the hang of your bunny’s moods.