Answers.Body: <p>I would recommend to use "Open Refine" for preparing your data!</p>
  
  <p>Open Refine (formerly: Google Refine) is an open source tool, running on your own computer, that allows to perform data manipulation as you need for any CiviCRM migration project.</p>
  
  <p>To get a quick idea how it works, you could take a look on these tutorial videos:</p>
  
  <p><a href="https://www.youtube.com/watch?v=yNccGtn3Wb0" rel="nofollow">https://www.youtube.com/watch?v=yNccGtn3Wb0</a> (Introduction 1 of 3)</p>
  
  <p><a href="https://www.youtube.com/watch?v=45EnWK-fE9k" rel="nofollow">https://www.youtube.com/watch?v=45EnWK-fE9k</a> (Introduction 2 of 3)</p>
  
  <p><a href="https://www.youtube.com/watch?v=5tsyz3ibYzk" rel="nofollow">https://www.youtube.com/watch?v=5tsyz3ibYzk</a> (Introduction 3 of 3)</p>
  
  <p>The third video will show how to "add column by fetching URL's" - this should deliver what you need!</p>
  
  <p>I have save hundreds of hours using OpenRefine during CiviCRM data migration projects. Of course, you will have a learning curve in using it. But this is one of the rare tools where you will save time already the first time you are using it.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-05T07:22:14.477Z
Answers.LastActivityDate: 2016-08-05T07:22:14.477Z
Answers.UserDisplayName: Detlev Sieber
Answers.Body: <p>Maybe a PHP script that uses DOM references to build an array of all the <code>&lt;li&gt;</code> items' names and link URLs, then cURLs each page and again uses DOM references to grab the contact info and add it to the relevant item in the array. There's probably a less procedural/more OOPy way to do it if that's your flavour. </p>
  
  <p>In the linked pages, you may be able to distinguish between the email address, web address, and bio with an if/elseif type of thing where if the contents of the A HREF contain "mailto:", that's the email; if it contains "http://" it's the web URL; failing both those conditions it's the bio.</p>
  
  <p>CSV is literally just comma-separated plain text where commas delineate columns and line-returns indicate rows, so if you can get your script to output "Name, email, web URL, bio (line return)" for each person in the database, you should be able to copy that text, paste it into a plain text document, and save it as CSV. </p>
  
  <p>Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T18:50:24.560Z
Answers.LastActivityDate: 2016-08-04T18:50:24.560Z
Answers.UserDisplayName: Sean MacGillivray
Answers.Body: <p>I've actually had good luck ("luck" being the operative word here) copying and pasting HTML tables in to a spreadsheet. You might need to paste it into a word processor document so it's in a table there first.
  I've done this with OpenOffice. YMMV with Microsoft Office.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T15:48:19.837Z
Answers.Text: Ugh. I just looked at the directory you're trying to work with. You might be able to automate that to some extent, but I'm not sure it's going to be less work than copying and pasting the data.
Answers.UserDisplayName: Dan O'Brien
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T15:43:00.387Z
Answers.LastActivityDate: 2016-08-04T15:43:00.387Z
Answers.UserDisplayName: Dan O'Brien
Body: <p>I'm organizing my data to prep for CiviCrm.
  One of the data banks we have is an online directory listing of hundreds of names, contact information and bios. NONE of this is organized into a spreadsheet. </p>
  
  <p>The information is 
  1. Hand coded into HTML
  2. Landing page has an alphabetical listing of hundreds of names that link to 
  3. A page in a separate digital directory with a html table with the contact information on one side and the bio on the left. </p>
  
  <p>QUESTION:
  How do I automate the scrapping of this information and pushing into a .CSV without having to do this one by one?</p>
  
  <p><a href="http://leonardo.info/led.index.html" rel="nofollow">Here is an example of the full director</a>y. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-05T21:48:13.713Z
Comments.Text: This isn't really a CiviCRM-specific question, as the same problem would apply regardless of what system you were importing into. You might get better responses by asking on a more general Stack.
Comments.UserDisplayName: choster
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-15T22:00:55.637Z
Comments.Text: These are all very helpful. I also looked at import.io which has a pretty simple interface. I do think, however, this is going to end up in a manual mode since the original data was not organized with any streamlined system. Thanks for all your answers.
Comments.UserDisplayName: Danielle Siembieda
Tags: <contacts><database><export>
Title: How do I scrape an old website into a .CSV file for data migration?
ThreadText: # How do I scrape an old website into a .CSV file for data migration?
  
  (Asked by: username_0 on 2016-08-03)
  
  I'm organizing my data to prep for CiviCrm.
  One of the data banks we have is an online directory listing of hundreds of names, contact information and bios. NONE of this is organized into a spreadsheet.
  
  The information is
  1. Hand coded into HTML
  2. Landing page has an alphabetical listing of hundreds of names that link to
  3. A page in a separate digital directory with a html table with the contact information on one side and the bio on the left.
  
  QUESTION:
  How do I automate the scrapping of this information and pushing into a .CSV without having to do this one by one?
  
  [Here is an example of the full director](http://leonardo.info/led.index.html)y.
  
  **Tags:** contacts, database, export
  
  > **Comments:**
  > **username_1**: This isn't really a CiviCRM-specific question, as the same problem would apply regardless of what system you were importing into. You might get better responses by asking on a more general Stack.  
  > **username_0**: These are all very helpful. I also looked at import.io which has a pretty simple interface. I do think, however, this is going to end up in a manual mode since the original data was not organized with any streamlined system. Thanks for all your answers.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2016-08-05. Score: 2
  
  I would recommend to use "Open Refine" for preparing your data!
  
  Open Refine (formerly: Google Refine) is an open source tool, running on your own computer, that allows to perform data manipulation as you need for any CiviCRM migration project.
  
  To get a quick idea how it works, you could take a look on these tutorial videos:
  
  <https://www.youtube.com/watch?v=yNccGtn3Wb0> (Introduction 1 of 3)
  
  <https://www.youtube.com/watch?v=45EnWK-fE9k> (Introduction 2 of 3)
  
  <https://www.youtube.com/watch?v=5tsyz3ibYzk> (Introduction 3 of 3)
  
  The third video will show how to "add column by fetching URL's" - this should deliver what you need!
  
  I have save hundreds of hours using OpenRefine during CiviCRM data migration projects. Of course, you will have a learning curve in using it. But this is one of the rare tools where you will save time already the first time you are using it.
  
  
  ---
  
  ### Answer by username_3 on 2016-08-04. Score: 1
  
  Maybe a PHP script that uses DOM references to build an array of all the `<li>` items' names and link URLs, then cURLs each page and again uses DOM references to grab the contact info and add it to the relevant item in the array. There's probably a less procedural/more OOPy way to do it if that's your flavour.
  
  In the linked pages, you may be able to distinguish between the email address, web address, and bio with an if/elseif type of thing where if the contents of the A HREF contain "mailto:", that's the email; if it contains "http://" it's the web URL; failing both those conditions it's the bio.
  
  CSV is literally just comma-separated plain text where commas delineate columns and line-returns indicate rows, so if you can get your script to output "Name, email, web URL, bio (line return)" for each person in the database, you should be able to copy that text, paste it into a plain text document, and save it as CSV.
  
  Good luck!
  
  
  ---
  
  ### Answer by username_4 on 2016-08-04. Score: 1
  
  I've actually had good luck ("luck" being the operative word here) copying and pasting HTML tables in to a spreadsheet. You might need to paste it into a word processor document so it's in a table there first.
  I've done this with OpenOffice. YMMV with Microsoft Office.
  
  > **Comments:**
  > **username_4**: Ugh. I just looked at the directory you're trying to work with. You might be able to automate that to some extent, but I'm not sure it's going to be less work than copying and pasting the data.  
  
  ---
  
  