I don’t think anyone was trying to say you’re a bad pet owner. However, since this is a house bunny forum, there are going to be people here advocating for bunnies to be kept indoors.
Personally, I would worry about leaving bunnies outside unattended and, especially, overnight. I’ve heard too many horror stories of what can happen to hutch bunnies. If your bunny is safe and secure where he is, then the only thing I would really worry about is extreme temperatures. Prolonged periods of extremely hot or cold temperatures are not good for bunnies, as well as sudden changes in temperatures (hard for them to adjust). So keep that in mind and have a plan in place. Perhaps it would be easier to transfer him using a carrier.
As Roberta suggested, you’re going to want to think about fleas. Even on a deck or balcony, fleas can find their way to a warm fuzzy animal. If you apply a topical flea treatment regularly, it can be easily managed. Revolution is the safest option. And never, ever use Frontline. There is a warning on the label about rabbits and we have had several cases on the forum where someone’s bunny had seizures and died after having Frontline put on them.