Hey there! New bunnies that are brought home should be left alone for the first week, minimal interaction is recommended because its a new environment for them and they need time to adjust, so this is probably why he has been nervous/scared as you’ve seen by his shaking.
As for rapid breathing.. I’ve found that one of my buns shakes while breathing because of multiple reasons, sometimes its because shes scared and other times its because shes hot. If she’s hot, then shes most likely flopped and panting at the same time. Her ears will be up but aren’t pushed forward, or they’ll be pushed back in a more relaxed position.
If shes standing/sitting up and panting/breathing fast, i look at the rest of her body language. Eyes wide, rapid breathing, ears straight up or pushed forward, is usually what she looks like when shes scared. Its pretty easy to tell what startles her, usually its my dog barking/howling or i accidentally knocked something against her cage.