I don’t disagree with your comment, (though I never served and personally never regretted my decision, a military environment simply isn’t for me), but based on a 2009 study that was referenced in this article from 2018, more people are disqualified on medical grounds than on fitness grounds.
It’s possible the ratio has changed in the past decade, I know that certain branches are now more lenient with waivers for various medical issues, but no matter how you slice it, the military is still out of reach for a ton of people.