Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What?s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered. 

 Nintendo announces new console but profit dives 
 
 TOKYO — Nintendo said on Monday it would release a new console next year to succeed its popular Wii, as it reported earnings slumped due to the strong yen and its 3D handheld player came too late to give a lift. 
 
 The video game giant said the new machine would be showcased at E3 Expo in Los Angeles in June. The move comes as Nintendo has struggled to reverse its sliding fortunes in an increasingly crowded market. 
 
 Nintendo said Monday that group net profit for the year to March fell 66.1 percent to 77.6 billion yen ($942 million), while operating profit slumped 52.0 percent to 171.1 billion yen. Sales eased 29.3 percent to 1.01 trillion yen. 
 
 However, the firm did say it expected sales to surge more than 40 percent over the coming 12 months. 
 
 The company said it suffered a serious setback as sales of both hardware and software fell in the United States, Europe and Japan. 
 
 Nintendo's release in late February of the DS3, the world's first video game console with a 3-D screen that works without special glasses, came too late to lift the results. 
 
 The company, which for many gamers is synonymous with its "Super Mario Bros" series, is facing a raft of challenges to its one-time domination of the market. 
 
 Traditional gaming rivals such as Sony with its Playstation, and Microsoft whose XBox has been a global hit, have been joined in the fray by smartphones and tablet computers, including Apple's iPhone and iPad which have taken a chunk of the gaming market. 
 
 Next year's new console will replace the popular Wii, which has sold 86 million units since its 2006 debut, but has seen a steady decline in recent years. 
 
 Sales of Wii hardware dropped 27 percent to 15.08 million units in the past fiscal year, with software revenue falling 11 percent. 
 
 It did not elaborate further on the new console. 
 
 A strong yen as well as price cuts also hurt earnings, Nintendo said. It lost 49.4 billion yen due to the strong yen as it increased the book value of foreign currency-denominated debts. 
 
 Nintendo needs a new boost to its slumping sales. 
 
 Global sales of Nintendo DS hardware tumbled 35 percent in the year to March from the previous year to 17.52 million units, excluding the latest 3D models. DS software sales fell 20 percent. 
 
 The company said however it expected profit to recover in the coming year. 
 
 It forecast net profit to rise 41.7 percent from last year to 110 billion yen. Operating profit is estimated at 175 billion yen, up 2.3 percent, and sales at 1.1 trillion yen, up 8.4 percent. 
 
 But Nintendo sounded a note of caution on future consumer spending after the March 11 earthquake that triggered a devastating tsunami along the northeastern coast of Japan. 
 
 "The aftermath of the Great East Japan Earthquake is unpredictable," the company said. 
 
 The Kyoto-based firm said it avoided direct damage to its factories and production network. 
 
 "However, it can be predicted that there will be indirect impact from individual consumption patterns or economic conditions in the future," it said. 
 
 Copyright © 2012 AFP. All rights reserved. More »
– Nintendo will follow up its popular Wii with a new console system next year, the company announced today. The new system, which IGN notes is codenamed Project Cafe AKA Wii 2, will be showcased at Los Angeles's E3 Expo in June. The announcement comes as Nintendo reported sagging earnings and sales, AFP notes. Despite the Wii's popularity, sales have been declining as Nintendo faces increased competition from other console systems as well as iPhones and iPads. IGN rounds up rumors about the new system: It could cost between $350 and $400; one of the names being tossed around is "Stream"; and it is expected to look like a modern Super Nintendo Entertainment System and be about the size of the original Xbox 360. The controllers will have integrated touchscreens, and the console will be able to stream games to each controller.