Remember a Musket has what 3000J of energy. That is 3000 J of energy into the enemy and 3000 J of energy going into you via the wood stock. Remember Newton's 3rd law.
So this actually really gets into the armour thing.
See, the way all this works is that a weapon is trying to do the target an
injury. A blade or spear point involves a certain amount of energy and then that energy is concentrated into a very small point, which means that it can do a major wound to the target. This is also why arrows have cutting heads.
Below a certain level of energy, the weapon cannot do this.
Part of what armour does is that it absorbs and also spreads out energy. It's easier to endure being punched hard than being stabbed with less actual energy involved in the stab, because the stab's energy is focused down to a point and so it can more effectively do damage.
When a musket fires, the energy and impact of the recoil is spread out between (for a right handed shooter) the left hand holding the stock and the right shoulder which is behind the stock, and in addition the stock is a very flat surface that effectively spreads out the force - it may be most of 3,000 joules, but the contact patch is dozens of square centimetres.
The
ball, meanwhile, is much smaller. It concentrates a lot of energy into a small impact point - much larger than the super-thin point of a broadhead arrow or a spear, but smaller than a punch and much smaller than the contact point of the stock on the shoulder.
So all that energy is concentrated in a very small space, and it defeats the ability of the skin to locally resist the impact. It breaks through and causes an injury - it delivers energy which causes the flesh damage as it absorbs the energy. It's less effective at doing that damage for a given amount of energy than a bladed weapon, because a blade can cut while a ball is rupturing, but the ball has so much energy that it can actually do significant damage in that way.
This is why I'd rather be punched (very hard) than shot (even with an air rifle) because the punch, while it has more force, is spread out over a greater area.