Not many people are experienced with baby buns on this site, so you might be better off finding a site that has breeders to give you advice. I think http://www.rabbitsonline.net/forums is where we normally advise, but hopefully a FL will pop in with more info.
One of my bunnies was the runt of the litter, and is now happy and healthy, but the breeder didn’t attempt to sell him until he was 10 weeks – even then he was tiny.
What breed are the parents? If both parents have the dwarf gene, then you can end up with a ‘peanut’ baby, that gets two copies of the dwarf gene, and will never survive past a few weeks. They often have bigger heads, and misformed back legs. They look signficantly smaller than the other babies. A peanut is different to a runt, runts have a reasonable chance of survival, whereas a peanut will never survive.
All you can do is what you’re doing i think – make sure he feeds from mum, top him up with extra feed if you can, and keep him warm and with his siblings.