Answer

First, because it is cumulative. Everything builds on what came before. So, if you fail to learn/understand one piece, then it is almost impossible to make sense of later parts of mathematics that depend on that piece.

Second, it is taught poorly, especially in early grades. People who become elementary school teachers often (know that they) don't really understand mathematics, and think it is less important for them to fully understand mathematics. Combining this with the first point, you can see why many people will, at some point, find mathematics to be hard

Third, over time, math textbooks get worse. To convince yourself of this, go find a copy of Courant’s 1937 book on calculus. In my opinion, this is the best calculus book ever written. It explains all of the key ideas and illustrates them with real applications. It teaches people how to solve hard problems. The “new math” of the 1960's replaced this approach with an emphasis on abstract theory (possibly inspired by the efforts of Bourbaki to do the same thing to math at the highest level of cutting edge research). This might have been appropriate for the miniscule fraction of people who would go on to become professional mathematicians. But it was a disaster for most students. The over-reaction was to remove most of the theoretical aspects, and replace them with … effectively nothing. This resulted in textbooks without adequate theory and without real applications. Instead, students got (and still get) drill-and-kill based on pattern recognition rather than understanding. And they got obviously artificial problems like “the melting snowman walking past the lamppost” that encouraged students to believe that calculus had nothing useful to offer.