I do think some rabbits perceive the broom as another rabbit. A broom is hairy, it moves, and it intrudes into the bun’s territory where it messes with the bun’s poop. Buns have territorial poops that they mark their area with. The broom tries to remove this poop, which must be regarded as an act of hostility. The fact that your bun starts to poop everywhere when you sweep would rather support this theory.
I had a rabbit who clearly saw the broom as his enemy. He would accept getting pet on the head with it though (the rabbit that gets groomed by the other rabbit is the dominant rabbit, so when the broom would pet my bun’s head, the broom was being submissive – in the eyes of my bun, that is).
As Dana says, the best thing is to try and sweep when your bun is in a different area. You can try letting your bun investigate the broom when it’s not moving, because that could perhaps help make your bun understand that the broom isn’t a live being. My bun still fought with the broom when it was hanging in its place in the hall, but that could’ve been because it smelled of my other bun (I had two buns and they were living in separate parts of my apt because they weren’t friends).