Why is it so rare to see new long running anime series these days? Around the early and mid 2000's we saw the spawning of a large collection of long running anime series, such as Naruto, One Piece, Pokemon, Yu-Gi-Oh!, etc., all of which still are continuing today in its original series, or through some spin-off. Nowadays, we are left with mainly 12 or 24 episode anime, most of which will never see a second season.

Why is it so rare to see new long running anime series these days? What caused the shift in the anime industry to a more "agile" (borrowing a term from software development for lack of a better word) production rate?

My first guess is that the producers base their decision for a long-running anime series on the success of its source manga. However, this is not entirely true, as we see Attack on Titan is a 25 episode series, even though I believe that its manga sales overtook that of One Piece at some point.
There are a variety of reasons for this. Some may apply and some may not.

The anime series catches up to the Source Material This is by far the most common problem, faced by long running Anime. Once the source material ends there are two options. Fillers or hiatus. 
Anime like Naruto and Bleach went the filler route. Bleach ended before the final saga which is still publishing. Naruto has finally ended but for some reason Anime is still prolonged by random fillers (Maybe to increase profit).
Shows like Fairy Tail and One Piece experimented. Fairy Tail abandoned the filler model and went on hiatus. Toei has screwed up One Piece with barely showing 5 minutes of new material and now again on fillers. So these long running shows have had their own problems and have overcome them only if the viewership and manga sales continue. 

Now why more anime are not in the same model and going at 250+ episodes?


There aren't just that many popular series which can be adapted for hundreds of episodes For a long running perennial series several factors have to align. Not only the series, but its other products should also be popular. Bleach, One Piece and Naruto had Mangaka's which constantly worked hard to put out new chapters and manga sales were good. Pokemon and Yu-gi-oh's trading card games were very popular. Their viewership numbers also didn't fluctuate. Take Hunter x Hunter for example. It had the potential to be right there with the big three, but due to inactive Mangaka its source material was finished way too soon. Even the remake could only adapt a couple extra arcs. More such examples are covered in Ryan's answer below.
The anime is adapted in a bad way. 
Kingdom has 470+ Manga Chapters. It is great source material and decent Manga sales. Its first season had such a bad animation that 90% of people who picked it up dropped it after second episode. Still it has over 70 episodes over 2 seasons. A call for reboot is there but I don't have my hopes up.
Tokyo Ghoul was given a second season where the writers completely went away from the source material. Even though Season 1 was received well its writers had made several unwarranted changes which led to many plotholes. (MadHouse's Parasyte though got it right. A great short Anime)
Seasonal Structure is less Risky. This is primarily the main reason from the departure from the perennial episodic structure from the big three. This method has lots of benefits with no obvious drawbacks. The studio only needs to commit to next season IFF it made profit the previous season. It also gives source material to continue ahead and some breathing room for the anime scriptwriters without lowering quality or pacing of the anime. The long running anime have many ups and down with the pacing. 
Tbh this isn't exactly new. Major for example ran over 6 Seasons consisted of 150+ Episodes between 2004-2010. 
JoJo's Bizarre Adventure has plenty of source material written from 1986-2004. But instead David Production went with a Seasonal model which is currently on part 3. So it is infact going to be a long running series.
Haikyuu and Kuroko No Basuke are a couple of other sports shounen which got subsequent seasons. KnK ended with season 3. Haikyuu is renewed for its 3rd season.
People prefer shorter anime This may or may not be true. But being from India and an Anime/Manga fan for over a decade, I didn't expect the sudden fad/growth of the Anime community. Naruto ending and its social network buzz seems to have given people a stimulus to pick Anime. Most of the people who ask me for recommendations want the Anime of lengths 24-25 episodes only. 


Tl;dr As the market became more money oriented and people moving on too quickly. The seasonal structure of Anime production gave the studios and production houses more flexibility to hedge their risks by producing more anime but only prolonging the release of the more profitable and popular titles. This also seem to orient with the requirements of the newer generation of fans who seem to prefer the shorter, high paced and high quality of anime.