What youre doing is very dangerous. Hes around a hormonal rabbit (your female) so hes going to act out hormonally even if he is neutered. You noted that he is already stressing her out and scaring her. If she grows afraid of him, she will never bond with him and she will always see him as something to be afraid of. You dont want that. Rabbits also mount and grip much, much harder than guinea pigs. He could easily break their thin arms, legs, spine, or even fracture their hips. You shouldnt be putting rabbits with guinea pigs anyway, as rabbits are so much stronger and communicate in very different ways that could seriously injure a guinea pig.
Separate your guinea pigs and your rabbits, but also separate your male from your female until she is spayed. You mentioned multiple rabbits; you need to keep everyone apart until theyre all neutered/spayed. Any hormonal rabbit in the area can easily cause even fixed rabbits to act out hormonally. Note, if scared enough, they may fight. If they fight, and youre not around, its possible that they will fight to the death.