It could just be that he is marking his territory now. If so, he may quit soon. Or if he’s really territorial, he may continue to do it.
What kind of cage does he have and what kind of litter box? If he is inside some sort of enclosure, make sure there is no bedding in it. If there is bedding anywhere besides the litter box, it is like offering the entire cage as a potty to him. Sometimes using a larger litter box helps. Some people will even move the food bowl into the litter area so that the bun has access to both food and hay. Usually when they eat hay they like to go, so if you have a hay rack hang it above the litter area too. When you clean up any poo, place it in the litter box. They can see and smell it and usually get the idea.
It is good though that he is at least peeing in the box. Some people have issues w/ their males spraying or peeing. If he isn’t fixed it, his litter box habits may improve if you have him neutered. And sometimes they are just never perfect. They can kick and scatter poo out of the box from time to time or have the occasional accident.