There seems to be quite a disparity in the ideal age to be successful at different sports. Tennis players seem, often, to thrive before they leave their teens, especially in the womens ranks; They're almost all done by 30. Good golfers barely get started by the age of 30. Baseball players may become "professionals" out of high school, but they rarely reach the major leagues before the age of 23 and statistically, most reach their peak at 27. Basketball players may start in the NBA at 18, but the cream of the crop is always over 21, and often much older. The NFL has draft rules keeping younger players out of the league, but 23 is a pretty good base age for beginning success in the league.
It seems the main difference is in the skills and development needed to play the different sports. Of the ones I listed, tennis and golf seem to be the outliers - tennis relies on speed, agility, and endurance, and does not require fully developed adult strength (especially for the women) - golf requires, more than anything else, accuracy. Football, Baseball, Basketball, seem to fit in the same group, as they all require more strength; Baseball again requires precision and accuracy that can only be developed over time; In the NFL, quarterbacks age more like baseball players than the other skill positions or defensive players.
These are only the sports I know the most about; I wonder what other sports age patterns are like. Soccer seems like it could have younger players succede, rugby probably not. Gymnastics and figure skating seem at the surface to be similar to tennis, but I think that is largely due to their amateur-olympics popularity.
Peoples bodies change throughout their lives, and people are better suited for different sports at different ages, though they normally need to have developed their skills over time (no jumping sports). I'd be interesting in seeing how much this is applicable to employment in non-physical professions.