Yes, I agree that timeouts and physical punishment are not going to work. You can use a “timeout” to break the cycle of a repetitive behavior, but this type of “timeout” is simply removing bunny from the situation. For example, they just won’t leave something alone even though you go over to stop them multiple times. I have found that removing them from that situation for awhile can help break the cycle and they will eventually decide to move onto something else. You can also combine this with redirecting them to something more appropriate. Simply locking them up in a cage when they do something “bad” isn’t going to register with them that they are being punished.
As for physical punishments, that will only teach your bunny to fear you. They won’t understand why they are being punished and they especially won’t understand that pooping outside the litter box is “bad”.
There are tons of resources out there for litter training. Keep in mind that some bunnies are going to be better than others when it comes to using the litter box. Setting him up for success is going to be your best bet.
And yes! You must get him neutered if you expect him to have any kind of consistency using the litter box. The urge to mark is just too strong when they aren’t neutered. We see it all the time. Someone is convinced their bunny can’t be litter trained. Then they get them spayed/neutered and TA-DA! Bunny is now using the litter box.