score:6
The answer depends very much on your degree of risk aversion, and on how much of your "like to be early" buffer you are willing to lose to traffic or other delays en route.
As Michael Hampton noted in a comment, the drive is almost entirely motorway, so depending on your driving habits and the traffic, you might be able to average 75 miles an hour (120 km/h) or you might not do better than 45 (72 km/h). So I'd probably allow six hours of driving time, plus time for breaks to refuel and whatnot.
If you are willing to arrive at 5:30 if the traffic is heavy, you could probably get away with leaving at 11:30. But if you are not, and being there at 4 is truly important to you, you should leave at least by 10, and if you are strongly averse to risk and/or the stress of worrying whether you'll make it, you might want to leave closer to 9.
Traffic on I-95 in Connecticut can be truly horrendous. It once took me something like 6 hours just to drive from Manhattan to New Haven, about 75 miles (120 km), but that was the Sunday after Thanksgiving and there was a particularly nasty accident on I-95. I was also on the road once immediately after a bridge collapsed, but I was not driving and I did not note the precise duration of the trip.
Horrific anomalies aside, I would leave more time if it is Sunday or Friday; three hours from New Haven to New York wouldn't be terribly surprising.
You might give yourself some flexibility by planning to take a little detour for a nice lunch or something like that, somewhere closer to the airport. If the traffic turns out to be bad, you can cut that short, or skip it altogether and go straight to the airport.
Upvote:2
If you really want to avoid the worst of the Connecticut and NYC traffic, or if you want to break up your trip in the middle, you could always use the Cross-Sound Ferry from New London, CT to Orient Point, NY (on Long Island.) The voyage takes about an hour and a half, and costs $55 (one-way) for a car & driver, $15 for each additional adult, and $6 for each additional child. Departures are at least once per hour during daylight hours, and sometimes twice per hour.
This route doesn't actually save you any driving time (Google Maps shows 2 hours from Barnstable to New London and 2 hours from Orient Point to JFK); and instead of CT traffic, you'll have to deal with inbound traffic on the Long Island Expressway instead. (Choose your poison.) But it might be a fun thing for the kids to do; if nothing else, it'll give you a break in the middle of the day to get some food and let the kids stretch their legs.