There are two related reasons.
The first is that feature film production, from the 1920s onward when film cameras became standardized, was entirely designed around working at 24fps. Cinematographers learned to exercise fine control over exposure, image quality and depth of field by balancing shutter angle (how long the frame exposes--literally how wide the gap or gaps in the film camera's spinning shutter is), f-stop (the width of the iris inside the lense), film stock (faster but grainier vs. slower but cleaner) and filters (at a minimum, neutral density filters, which reduce light transmission to the film without altering color). There are some very complicated tables and a lot of rules of thumb built around balancing all these elements, and altering the frame rate throws still another problem into the mix. Meanwhile, it decreases the amount of footage that can be shot before the film magazine must be switched by 20%, and bumps up the cost of film stock by just as much. Timing, editing and sound sync processes were all designed around 24fps as the desired output. 896
This adds up to a lot of infrastructure and know-how optimised for 24fps, in a high-pressure production environment where time is money, and small mistakes can add up to even bigger money.
But it was (and is) certainly possible to shoot at higher frame rates. It's simply not worth it, for an additional reason. If the goal is to gain a higher quality image on tv, higher frame rates can actually work against you. I went into this in an answer about the "Soap Opera Effect," but the bottom line for this question is that the "judder" introduced by duplicating those frames isn't usually very noticeable, but the increased overall "sharpness" of motion at higher frame rates is.
And now we're getting into opinion, but I believe it's partly cultural, and partly about how our lower vision system translates speed of motion into perception of safety or danger. 24fps feels more like a waking dream than a real situation. This isn't what we want for sports or videogames, but it's exactly what we want to wrap our fiction in. 862