A bit of column A, a bit of column B.
Yes, 50% child mortality skews life expectancy statistics heavily, but any 40 year life expectancy estimate is clearly filtering out at least some portion of childhood deaths. By our best estimates: of the 48% of people who survived age 10, slightly less than half were dead by 45. Of those who clear 45, less than half reach 65.
Those early deaths aren't driven by "inferior physiology", but disease and malnourishment (as the previous commenter noted). It was possible to live into your 80s, but you had to be very, very lucky to pull it off.
Medieval peasants worked more and harder hours than modern people.