Definitely not too long of a car ride! Mine have visited my parents a few times and they are more than 7 hours away. I give them some hay for the trip, and they eat a little of it. I generally don't provide water as it makes a mess, and they only drink once or twice a day as it is.
Is he a particularly sensitive and nervous rabbit? If so, he might be better off at home, though my girl is fairly nervous of new things and settles down quickly at my parents' house. In your situation, I'd prefer to have him with me if it's not too much trouble, but I'm sure he'd be fine at home too.