Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. 

 "It's almost like you're going so far out of your way and almost doing back flips and cartwheels to defend Trump. It's just a turn-off at this point." [Caller to Rush Limbaugh, March 31] 
 
 Conservative media's "Trumpkins"-fueled civil war has spared few victims. Just ask Rush Limbaugh, who continues to take on fire as he stumbles his way through the right-wing media's divisive Primary Season from Hell. 
 
 Routinely condemned for not calling out what Republican critics see as Donald Trump's brand of faux conservatism, Limbaugh continues to fish around for a middle ground. The host seems anxious to defend Trump from attacks, but also wary of offending his legion of listeners, who see the front-runner as a fraud, and see Limbaugh as a hypocrite for playing nice with him. 
 
 Torn between the allure of what's popular (Trump) and abandoning everything Limbaugh's said about how he defines conservatism over the last 30 years, Limbaugh now often finds himself in no-man's land. 
 
 "Every day of Rush's show now feels like an exercise in strained, compulsory quasi-neutrality, which amounts in practice to him defending nearly everything Trump says and does but mixing in some praise for Cruz here and there just to make sure he's got his footing on the tightrope," wrote the Hot Air blogger known as Allahpundit. 
 
 In other words, Limbaugh's playing defense, a mode that most talk radio hosts despise. 
 
 Have there been previous primary squabbles, and has Limbaugh been at odds with his famously like-minded listeners in the past? Of course. But as a rule, the conservative media world over the years hasn't been known as the home of freewheeling and raucous partisan debate within the GOP, pitting Republicans against Republicans. Instead, it's been known to be an amazingly disciplined echo chamber that directs its fire outward toward Enemy No. 1: Democrats. 
 
 The current primary battle is the most bitter in recent memory. It's also threatening to implode the Republican Party -- and to a degree, the entire conservative movement as we know it -- as Trump angles to secure the party's nomination while breaking free from core beliefs that have been considered sacrosanct for generations by Limbaugh's listeners. 
 
 The host, meanwhile, has become a piñata, as conservatives line up to take whacks at him in a way we've never really seen before. 
 
 In a scathing critique of Limbaugh's support of Trump, The Wall Street Journal's Bret Stephens lamented that conservatives "used to have the good sense to dismiss" unserious candidates like Trump "as eccentrics, lowlifes or clowns." 
 
 Michael Gerson, former speechwriter for President George W. Bush, complained, "Through a long series of controversies, Limbaugh has excused Trump's narcissism and bluster as an endearing 'schtick.'" 
 
 Then he lowered the boom: "Populist anti-intellectualism, on the rise at least since Sarah Palin, has culminated in Trump. It is the passing of conservatism, even if Limbaugh baptizes the dead." 
 
 At The Daily Caller, Matt Lewis nicked Limbaugh for "abdicating" his "responsibility" and not having the "intellectual honesty and moral courage" of local Wisconsin talk show hosts who stood up to Trump. Lewis blamed Limbaugh and other nationally syndicated hosts for feeding the Trump beast last year to the point where he now may be unstoppable. 
 
 "Limbaugh is a mind-numbing, frustrating hypocrite," wrote longtime listener Travis Hale at The Hill. "His tacit endorsement of Trump, now occurring daily during his show, is almost impossible to understand." 
 
 You don't get the sense that these brushback pitches are coming from Ted Cruz diehards who are furious with Limbaugh's partisan refereeing. Some of the missives are coming from people who are puzzled that someone they looked up to for so many years "to be our voice of conservative reason" is revealing himself to be a weather vane this primary season, pointing whichever way the (Trump) wind blows. 
 
 That includes some of Limbaugh's Dittohead listeners, who are chewing him out, too. "I believe that most of the time you stimulate my intelligence, but today I feel like you're insulting my intelligence -- and that of many other people," a caller recently lamented to Limbaugh. 
 
 In response, Limbaugh is alternately defending himself and periodically bowing out of the Republican free-for-all. 
 
 Retaliating against conservative commentators who have branded the talker a "sellout," Limbaugh returned fire last month with his own accusations of hypocrisy: "They turn to me as a source of blame for the fact that Trump hasn't been taken out, despite the fact that they've tried. But I don't see where they've endorsed other people. I don't see where they have actually engaged in the behavior they demand that I engage in." 
 
 When controversy erupted after Trump's campaign manager was charged with simple battery after grabbing the arm of a reporter, and after Limbaugh was seen as defending the Trump campaign, the host announced that the topic was suddenly off-limits. "I have determined here that because tensions are so tight, everybody is wound up to such a feverish pitch here, that no matter what I say it is misunderstood and is not helpful," he announced. 
 
 Lamenting the state of the GOP primary season, Limbaugh conceded the arm-grabbing incident might not be the only one where he'd have to "muzzle" himself: "Because it's apparently impossible to be correctly, properly, understood." 
 
 For someone whose entire career has been based on lies and misinformation, there's something wonderfully fitting about Rush Limbaugh silencing himself because his fans and conservative media just aren't listening properly and he just can't be understood.
– Liberal watchdog Media Matters has an email that it says shows how the honchos at Fox News order news to be slanted: It comes from Fox exec Bill Sammon, who sent it to news staffers on the subject of global warming: "...we should refrain from asserting that the planet has warmed (or cooled) in any given period without IMMEDIATELY pointing out that such theories are based upon data that critics have called into question. It is not our place as journalists to assert such notions as facts, especially as this debate intensifies." The email went out after a Fox reporter noted a UN finding that 2000-2009 was going to be "the warmest decade on record," and it had an immediate effect on subsequent reports. At Salon, Alex Pareene writes "even by the low standards of partisan pseudoscience, this memo is horrific." There's debate about whether humans are causing the planet to warm, he writes, but the "warming itself is a proven fact."