Ok so I decided to do some bonding sessions between Rufus, Bugz and Anya. But they had not been going well so I stopped and gave them a break. I found a lady that does bonding and we packed up and all went there. It started well but as the day went on it went south. Rufus became agressive that lead to Bugz being defensive and fights broke out. We tried a few alternatives but so far nothing seems to be working. We then tried to see who was the agressor and the issue, I thought it would be Bugz, however when we tried to bond him to another rabbit (one they had in at her place) he accepted this rabbit with little to no fuss, Anya likes everyone and Rufus tried to kill them all. I then spoke to their animal behaviour person and she said that this is not uncommon for an animal that has been in a bad situation as he is protective of me. When I was not in the area then he was Ok (not great but ok) as soon as I stepped into the area he could come and check me out and then attack other rabbits that he believed where going to take me away from him 
After talking to them and my vet we decided that he might as well just stay friends with the cat (who he is in love with!) and not try and bring another bunny in or stress everyone out more by trying to bonded them which may never happen. WE are not sure how much longer Rufus has with his mouth being such a mess so we figure that the best thing to do it keep doing what I am doing however I might change things around so they all have more room as out of the cage time is a little less then I would like due to having to do it in shifts.
The other thing I was going to say was that Rufus had his teeth done again and they are much better however he did lose one and the vet gave it to me but the vet was not worried as it was one of the ones that lead down to a bad abcess and also was causing jaw issues so the fact that it came out and she did not have to remove it was a plus money wise.