Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. 

 What Lindsay Lohan Can Learn From Lil Wayne in Prison 
 
 Email This As anyone who even casually follows celebrity crime knows, there can be a Grand Canyon-sized disparity between the sentence a star receives and the actual time served. Perennial court-attender and 'Herbie Fully Loaded' actress Lindsay Lohan was recently As anyone who even casually follows celebrity crime knows, there can be a Grand Canyon-sized disparity between the sentence a star receives and the actual time served. Perennial court-attender and 'Herbie Fully Loaded' actress Lindsay Lohan was recently sentenced to 90 days in jail for violating her probation in a DUI case. Rob Shuter already discussed Lohan's plan to avoid prison , but if she does actually serve part of her sentence, she's going to need help. Put down the 'Just My Luck' Blu-Ray and watch 'The Wire,' Lindsay. Prison ain't easy. The parallels between Lohan and Lil Wayne , who is currently serving a one-year sentence for felony gun possession, may not be immediately apparent, but here are a few tips the actress can learn from the superstar rapper. 
 
 Record a Song Behind Bars 
 
 Smuggle Contraband to Listen to The Classics 
 
 Get Diesel 
 
 Charge Guards to Watch Her 
 
 Receive Celebrity Visitors 
 
 Get a Cushy Job 
 
 
 
 Write to Your Fans 
 
 -- One day after Drake released 'Light Up,' his collaboration with Jay-Z, the rapper's team had Wayne record a new verse from his current home at Riker's Island. While the verse was middling in both audio and lyrical quality, it was still light years ahead of previous rappers' attempts to keep their careers going from behind bars. Lohan is no stranger to music; her 2004 debut album, 'Speak,' and its follow-up, 2005's 'A Little More Personal,' cumulatively sold over 1.5 million albums. Surely, prison will give her new lyrical inspiration to augment future classics like 'Confessions of a Broken Heart' and 'Bossy.'-- In May, Weezy was caught was with an unauthorized MP3 player and headphones smuggled in the cell of a fellow inmate. In certain jails, prisoners are allowed to purchase radio and MP3 players with headphones, but only those that have been approved and sanctioned by the jail. You have to assume, though, that there's at least one guard whose daughter is a 'Freaky Friday' fanatic that's going to bend the rules a bit so Lohan can study 'Paris' a few more times for inspiration.-- In April, popular hip-hop site Rap Radar posted an interview with rapper/Cash Money CEO Bryan "Birdman" Williams from DJ Scoob Doo's 'Godfather' DVD, in which the emcee discussed Wayne's (presumably exaggerated) workout schedule: 15,000 sit-ups and 20,000 push-ups a week. If Lohan were to miraculously serve her entire sentence, that's 180,000 sit-ups and 240,000 push-ups -- AKA enough to take care of even the burliest of post-prison paparazzi.-- A female former guard at Riker's Island was fired earlier this year for allegedly entering the men's ward for a quick peek at Lil Wayne. While the guard contends she was there to see a colleague, it didn't stop the Department of Corrections from showing her the cell, er, door. Regardless of whether the guard was an overambitious fan or victim of a cover-up, this should inspire Lohan to make the easiest cash possible in prison: charge guards to enter the female ward to watch her.-- When Fat Joe and Diddy made separate visits to the rapper in prison, music mags and sites everywhere salivated at the thought of secret discussions and the sharing of info and advice. In the pop culture world, nothing is more interesting to celebrity tabloid fans than when one tabloid regular visits another in prison. I have visions of a delusional Paris Hilton telling Lohan, "It's not so bad. They bring in a wardrobe for you to pick from, your meals are cooked by executive chefs and 8PM every night is champagne time."-- Anyone who has followed Wayne's career knows that the rapper is a workaholic, frequently entering his mobile studio at all hours of the night to record more verses. At Riker's, Wayne is a Suicide Prevention Aide , socializing with troubled and at-risk inmates for 50 cents an hour. Lohan should consider taking up employment pronto. May we suggest teaching acting classes to young inmates? You think 'I Know Who Killed Me' earned a whopping 8% on Rotten Tomatoes because of Julia Ormond?-- While Twittering from jail is still verboten, the rapper recently launched Weezythanxyou.com , a new website where the rapper, through letters written to his record label, can thank his fans, discuss sports and chronicle his everyday life. The site also features a countdown of Wayne's expected release date (113 days and counting). Lindsay needs to jack this idea pronto. With all the constant chatter about her, why not have one spot where she can shun the hyper-stylized, glossed-up rhetoric of publicists, lawyers and agents and actually write what prison is like and how she feels? At this point, humanization may be the quickest route to salvation. Either that, or someone just smuggle 'Da Drought 3' into her cell.
– As Lindsay Lohan’s prison date approaches, she might want to take a look at Lil Wayne—currently serving a year for gun possession—for some tips on how to survive the slammer. Jason Newman offers seven Wayne-inspired ideas on PopEater: Visit with friends: Celebrity friends, that is—just like when Fat Joe and Diddy visited Wayne. Can’t you just see “a delusional Paris Hilton telling Lohan, ‘It's not so bad. They bring in a wardrobe for you to pick from, your meals are cooked by executive chefs, and 8pm every night is champagne time.’” Make some money: A female guard was fired for allegedly sneaking into the men’s ward to see Lil Wayne. Linds can capitalize on this idea by charging guards to watch her sit around in her cell. Make some money, part 2: Lil Wayne makes 50 cents an hour counseling fellow inmates at risk of committing suicide. LiLo could try to get a job of her own, like “teaching acting classes to young inmates. You think I Know Who Killed Me earned a whopping 8% on Rotten Tomatoes because of Julia Ormond?” For the complete list, including a few suggestions to boost her musical career, click here. Or to see LiLo licking a gun in the promo poster for her new movie, click here.