If Bandit stays near and lets your brother pet him when he’s lounging, then I wouldn’t think that he’s particularly bothered. If Bandit was disturbed by your brother’s behaviour, I’m sure that he would move somewhere else as he has the ability to. While it is rude of your brother to disrespect your request like that, I don’t really think he’s doing anything wrong. He’s not forcing himself on Bandit (stopping him from escaping or cornering him etc). A half hour or so interruption in the middle of the day isn’t going to make him grumpy for the rest of the day or undo your bonding work. If he was trying to grab him or pick him up etc I would be concerned, but if all he’s doing is offering attention and pets, then I don’t see too much cause for concern.
Obviously my pair also sleep a lot during the day, but they don’t have a long solid sleep, they’ll come down for hay and go back up to sleep, then if they think I have something for them they’ll come down to beg and for a pet or two then go back up to sleep and so on. I think it’s pretty natural for animals like rabbits to have shorter sleep periods over that time when they spend the day sleeping. They don’t just sleep for 6-8 hours, it’s an hour or two, then a short burst of activity, another hour or two etc.