You should keep your bunny since you love him so much! I wondered about this as well since I’m not working full-time right now and have a lot of time to supervise the bunny’s playtime out of his cage. But other times in the past I’ve worked 60 hour weeks, and sometimes will be out of the house for 14 hours at a time. Today was actually the first day I left him alone for a solid 13 hours. I thought he’d go ballistic when I let him out since it had been so long, but he acted normal. I don’t even think he noticed.
As long as the rabbit cage is sufficiently large and he has lots of stuff to do in there he should be fine. Mine is still getting lots of outside time (only about 20 minutes in the morning), but now that I’m home at night I can have him out for at least 3 more hours before I go to sleep. And that’s just for the long days, on regular days it doesn’t have to be that limited. If the bunny has one of those tri-level cages and can run around in it, being cooped up all day isn’t the worst thing.