Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What?s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered. 

 Target strikes back at Lady Gaga, criticizes singer for backing out of deal over gay rights 
 
 Ceneta/AP Lady Gaga, stauch LGBT supporter, is walking away from a deal with retail giant Target. 
 
 Take our Poll Bad Romance What do you think about Lady Gaga ending her deal with Target? Good for her! Way to take a stand. She's just creating drama, as usual. Mixed feelings. Related News Articles 
 
 Lady Gaga axes deal with Target over anti-gay donations 
 
 Looks like Lady Gaga has landed in Target's bullseye. 
 
 
 
 The retail-giant slammed the multi-platinum pop star on Wednesday after reports emerged she had ended their professional partnership over Target's refusal to cease controversial political donations. 
 
 
 
 "We were very surprised and disappointed by the statements made by Lady Gaga's organization related to her partnership with Target," the Minneapolis-based company said. "Target remains committed to the LGBT community as demonstrated by our contributions to various LGBT organizations, our recently established Policy Committee to review our political giving and our respectful, inclusive workplace environment." 
 
 News that Gaga had terminated her relationship with the superstore was first reported in The Advocate on Tuesday, which cited a source close to the singer. 
 
 
 
 "She and Target didn't see eye to eye on Target's policy of political donations and how they affect the LGBT community," the source said. 
 
 
 
 A representative for the singer confirmed the news in a statement on Wednesday, revealing that the two sides "came to a mutual decision to end their overall exclusive partnership a few weeks ago." 
 
 
 
 Gaga and Target had been negotiating a deal that would have put an exclusive version of her upcoming CD "Born This Way" on sale in the big box stores in May. But after her fans voiced their reservations, Gaga – an outspoken activist for gay rights - reportedly reconsidered. 
 
 
 
 The LGBT community was very critical of Target's $150,000 donation to MN Forward, a political action committee that supported Minnesota Republican gubernatorial candidate Tom Emmer. A staunch opponent of same-sex marriage, Emmer failed to win the state's top job. 
 
 
 
 "Part of my deal with Target is that they have to start affiliating themselves with LGBT charity groups and begin to reform and make amends for the mistakes they've made in the past … our relationship is hinged upon their reform in the company to support the gay community and to redeem the mistakes they've made supporting those [anti-gay] groups," Gaga told Billboard last month. 
 
 
 
 While the specifics surrounding Gaga's decision to walk away from the negotiations remain murky, E! Online reported that Target refused to promise it would never donate to anti-gay organizations or politicians again. 
 
 
 
 As of Wednesday evening, Target.com still featured the deluxe version of "Born This Way" for pre-order on its website. It included the standard album, three new songs and five remixes.
– Target is blasting back at Lady Gaga who dumped a business deal with the discount retailer because she says the corporation didn't live up to promises to support the gay and lesbian community. "Target remains committed to the LGBT community as demonstrated by our contributions to LGBT organizations, our Policy Committee to review our political giving, and our inclusive workplace environment," Target said in a statement. The company came under fire last year for a $150,000 donation to a political action committee supporting a failed Minnesota gubernatorial candidate opposed to gay marriage, notes the New York Daily News. Gaga planned to release a special edition of her upcoming CD Born This Way in Target stores, but only if the company made "amends" for past political actions. E! Online is reporting that the deal broke down because Target refused to promise that it would never donate to anti-gay organizations or politicians again.