Which ferry ports are best in France for a family travelling as foot passengers?

My husband and I and our little one would like to go over to France for the week, but don't relish the thought of having to drive on the other side of the road (not for our first holiday with a new baby anyway). Which ports are best for direct access from the ferry, and have the best attractions/rail connections?