Not knowing what kind of surgery your little Fox had, I will share my experience with my Olaf who just had his neuter on Monday. I was told to keep him subdued and on one level, as you were. The tech also told me to keep him in his cage for at least three days. If your bun had a more complicated surgery, the “quarantine” period can be even longer so I would check with your vet if that is the case.
He was sleepy and uncomfortable for the first few hours after I brought him home but perked up before bedtime. The next day he seemed exactly like his old self (greeting me at the cage door, eager for treats, etc.) I kept him in the cage all day and decided to let him out in the x-pen the following morning. He did some binkies and seemed thrilled to be free. I, too, worried it might be too much but in many ways rabbits (and all animals) have a good idea of what is too much for them. I did wait another day, however, to reinstall his second floor — just in case.
I hope this helps and suggest calling your vet to get specific instructions for Mr. Fox. I hope he is feeling better and can binky to his little heart’s content soon! Good (((vibes))) and hoppy juju.