It is very easy to become frustrated when litter training, but i think the key is repetiton and consistency!
Have you tried restricting his play area? Maybe an exercise pen connected to his cage, that way you can cover the floor with a blanket and put a litterbox in the play area and his cage?
For the flipping of his litterbox, if his cage allows drill two holes near the top of the litterbox and affix it to the cage using zipties, or use duct tape? Be creative! My boy used to move his around and make a mess but finally figured out it wasnt a toy when it wouldnt budge.
Its going to take some time and patience to create a good litter habit. As much as it sucks, pick up any poos, soak pee with tissue and put it in his litterbox.
Another thing that works for my bunny, is praise. If you find him in his litter box pooping or peeing, even eating hay to begin with! Praise him and give him a small treat. You could also try clicker training!
Im still trying to get my bun to associate a clicker with reward for good behavior.