You need to be careful with holland lops outdoors. They are not really outdoor rabbits unless you live in a very monotoneish climate, or bring them in half of the year, in which, there would be no reason not to keep them inside all year around. They have to be in temperatures no lower than 40 and no higher than 80. A temp around mid sixties to low seventies is best. Remember also that rabbits can go into shock or become sick if you move them between a 10 degree difference (example: if its 40 outside and you bring him into your 70 degree house, he can go into shock, overheat, have a stroke, ect because the temp change is too great)
You mentioned gardeners, what do they do? If they spray any chemicals around, or put down any chemicals, yes, your rabbit will be in danger. If the gardeners just trim your plants and pull up weeds, your rabbit should be fine
Since he will be outdoor, he will be lonely and could be scared at night. Getting a friend is your best bet, as previously mentioned. Do note, however, that the two cannot live together until atleast one month after they are both spayed/neutered, and will need to be bonded before living together. This can take several months, but it is one of the best ways to ensure your rabbits comfort. Note, that rabbits can die from stress
Since they will be outside, you also have to keep them updated on their vaccines. The hutch must be cleaned every day or two at the latest to avoid bugs and infestations inside of it. The hutch must have appropriate protection around it to avoid predators from getting in. There must be appropriate amount of room on the ground for the rabbit to exercise. The hutch must be off the ground to help protect it and your bun from various things, for example: flooding inside of the hutch during a rainstorm. Half of the hutch must be completely covered so the rabbit can hide. Youll also have to make sure the rabbit cannot dig out of the hutch area, so youll need to have something underground to prevent the buns from digging out and running in. Likewise, youll need this so predators cannot dig their way into the hutch area