I want to download a list of web pages. I know wget can do this. However downloading every URL in every five minutes and save them to a folder seems beyond the capability of wget.
Does anyone knows some tools either in java or python or Perl which accomplishes the task?
Thanks in advance.
Write a bash script that uses wget and put it in your crontab to run every 5 minutes. (*/5 * * * *)
If you need to keep a history of all these web pages, set a variable at the beginning of your script with the current unixtime and append it to the output filenames.
How to download a webpage in every five minutes?