Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. 

 Meghan McCain took part in a new public service announcement promoting awareness of the dangers of skin cancer. In order make the point that being outside without sunscreen is like being naked, the provocative video features all of the female participants suggestively appearing naked. However, such a suggestion about McCain was enough to make Glenn Beck lose his lunch . . . repeatedly. 
 
 Beck’s radio show often leans more towards being comical, but already Meghan’s mother, Cindy McCain, made it clear that she’s not laughing. Beck just returned from Israel and Beck claimed he was starting to feel sick. Beck’s radio sidekicks wondered if it was the flight or the Middle East water that may have harmed Beck, but any time they mentioned the name “Meghan McCain” Beck would simulate more vomiting. Beck admitted the shots of Meghan in the video were “luscious.” He joked that maybe he was getting sick from too much of a good thing or maybe it was because “just try to imagine John McCain naked with long blonde hair.” 
 
 Before Beck’s review, Meghan confirmed that she was not actually naked for the filming of the video. And now after Beck’s commentary, Cindy has responded: 
 
 I’m so glad Glenn Beck is leaving Fox. Enough vitriol and hate. Glenn you are no rodeo clown. They are decent and nice. You aren’t. 
 
 It seems Beck’s advice about how Meghan should wear a burka from now on to be extra safe didn’t go over too well. Regardless, the message Beck received from the PSA was to “put some extra clothes on,” so whether the taunting was too mean-spirited or not, there is definitely heightened awareness now for the dangers of being “naked” in public. 
 
 Listen to the clip below from The Glenn Beck Program: 
 
 
 
 (h/t) 
 
 Have a tip we should know? tips@mediaite.com 

 [kyteembed s=”1331556″] 
 
 Glenn’s certainly proven to be very sensitive. He cries all the time, he complains about his vegan diet, he whines about the temperature of the studio. But as far as we know, very little actually causes him to vomit – just look at how he handles all that crap he eats all day! But if anything could turn his stomach, it’s naked videos of Meghan McCain. “May I just say if you want to talk about essential, luscious, delicious, yummy, just… just a delectable appetizer for the main course, it would have to be Meghan McCain’s naked commercial. Oh. My stomach, hang on. I might have eaten something bad.” 
 
 You can watch the video – an anti-skin cancer PSA – below: 

 Meghan McCain bared nothing more than her shoulders in a newly released skin cancer PSA, but radio host Glenn Beck saw it differently. 
 
 Beck, who just returned from a trip to Israel, joked on his show Wednesday that he wasn’t feeling well. Was it the water, his co-hosts asked? The bumpy trip back? Or was it because he had just viewed Meghan McCain’s PSA? 
 
 Beck wasn’t sure, he said. But every time McCain’s name was uttered, he grabbed a bucket and pretended to lose his lunch. “Oh, I got something bad. Might be that Meghan McCain,” he joked with co-hosts Stu Burguiere and Pat Gray. Between bouts, he added: “When I look at the picture, I think to myself, I get the point, put some extra clothes on.” 
 
 Both of McCain’s parents battled skin cancer, and Cindy McCain was quick to tweet her thoughts about Beck’s reaction. “I'm so glad Glenn Beck is leaving Fox. Enough vitriol and hate. Glenn you are no rodeo clown. They are decent and nice. You aren't,” she tweeted. 
 
 Here’s the PSA featuring Meghan McCain that prompted Beck’s teasing:
– Meghan McCain recently showed some skin in a PSA for skin cancer awareness (watch it here), which led to an eight-minute pukefest yesterday on Glenn Beck's radio show. Beck pretended to vomit every time McCain's near-nakedness was mentioned, while saying things like, "Put some extra clothes on ... Has she thought about, like, a burka? Just so she's extra safe?" Enter angry mama Cindy McCain. Politico and Mediaite note that she swung back with a tweet: "I'm so glad Glenn Beck is leaving Fox. Enough vitriol and hate. Glenn you are no rodeo clown. They are decent and nice. You aren't."