I’m going to go the other way on this. My opinion: As long as they are receiving adequate exercise each WEEK, I think this system might work. It is not the ideal, but I think I might be a good compromise. Sometimes we need to make sacrifices and do the best we can and I think they would all receive enough exercise. If you were to do this tho, I’d recommend that the other rabbits have an xpen attached to their cages for some safe space on the “off days”.
Also, keep in mind that rabbits can fight and or get pregnant thru the bars of a cage/pen. Are they all caged in the same room? The free roaming bunny must not be able to get within 3 inches of any other bunny. In addition to this, you should have a cover over the xpens so there is no risk of a bun hopping the pen.
Edited to add: Now that I am thinking about it, Ibelieve I remember Binkybunny saying she did something similar to this when she had a pair and a single bun. They switched exercise/free roam days, but the penned space was always very, very large to accomodate the others.
Two of your bunnies are of age to be neutered, correct? What is your timeline? Are you having issues finding a vet or is there something else in the way?