How to make LWP and HTML::TableExtract spitting out CSV with Text::CSV
first of all: sorry for being the novice. I am just beginning with Perl. So  don´t be rude with me if i ask beginner-questions. 

This all is a great learning curve for me.

first of all - this is a great community and a very very nice place to be! I love this great place. Today, one last question regarding a parser... I am currently working on a  little parser.

i have had very good results with the first script! This was able to run great! 
It fetches the data from the page: http://192.68.214.70/km/asps/schulsuche.asp?q=n&a=20
(note 6142 records) - But note - the data are not separated, so the subequent work with the data is a bit difficult. 

it spits out the data like this: 

    lfd. Nr. Schul- nummer Schulname Straße PLZ Ort Telefon Fax Schulart Webseite
    1 0401 Mädchenrealschule Marienburg, Abenberg, der Diözese Eichstätt Marienburg 1 91183  Abenberg   09178/509210  Realschulen  mrs-marienburg.homepage.t-online.de 
    2 6581 Volksschule Abenberg (Grundschule) Güssübelstr. 2 91183  Abenberg   09178/215 09178/905060 Volksschulen  home.t-online.de/home/vs-abenberg 
    3 6913 Mittelschule Abenberg  Güssübelstr. 2 91183  Abenberg   09178/215 09178/905060 Volksschulen  home.t-online.de/home/vs-abenberg 
    4 0402 Johann-Turmair-Realschule Staatliche Realschule Abensberg Stadionstraße 46 93326  Abensberg   09443/9143-0,12,13 09443/914330 Realschulen  www.rs-abensberg.de 

But i need to separate the data: with Commas or someting like that! 

And i have a second script. This part can do the CSV-formate. i want to ombine it with the spider-logic. But first lets have a look at the first script: with the great spider-logic. 

see the code that is appropiate: 

     #!/usr/bin/perl
        use strict;
        use warnings;
        use HTML::TableExtract;
        use LWP::Simple;
        use Cwd;
        use POSIX qw(strftime);
        my $te = HTML::TableExtract->new;
        my $total_records = 0;
        my $suchbegriffe = "e";
        my $treffer = 50;
        my $range = 0;
        my $url_to_process = "http://192.68.214.70/km/asps/schulsuche.asp?q=";
        my $processdir = "processing";
        my $counter = 50;
        my $displaydate = "";
        my $percent = 0;
    
        &workDir();
        chdir $processdir;
        &processURL();
        print "\nPress <enter> to continue\n";
        <>;
        $displaydate = strftime('%Y%m%d%H%M%S', localtime);
        open OUTFILE, ">webdata_for_$suchbegriffe\_$displaydate.txt";
        &processData();
        close OUTFILE;
        print "Finished processing $total_records records...\n";
        print "Processed data saved to $ENV{HOME}/$processdir/webdata_for_$suchbegriffe\_$displaydate.txt\n";
        unlink 'processing.html';
        die "\n";
    
        sub processURL() {
        print "\nProcessing $url_to_process$suchbegriffe&a=$treffer&s=$range\n";
        getstore("$url_to_process$suchbegriffe&a=$treffer&s=$range", 'tempfile.html') or die 'Unable to get page';
    
           while( <tempfile.html> ) {
              open( FH, "$_" ) or die;
              while( <FH> ) {
                 if( $_ =~ /^.*?(Treffer <b>)(d+)( - )(d+)(</b> w+ w+ <b>)(d+).*/ ) {
                    $total_records = $6;
                    print "Total records to process is $total_records\n";
                    }
                 }
                 close FH;
           }
           unlink 'tempfile.html';
        }
    
        sub processData() {
           while ( $range <= $total_records) {
              getstore("$url_to_process$suchbegriffe&a=$treffer&s=$range", 'processing.html') or die 'Unable to get page';
              $te->parse_file('processing.html');
              my ($table) = $te->tables;
              for my $row ( $table->rows ) {
                 cleanup(@$row);
                 print OUTFILE "@$row\n";
              }
              $| = 1; 
              print "Processed records $range to $counter";
              print "\r";
              $counter = $counter + 50;
              $range = $range + 50;
              $te = HTML::TableExtract->new;
           }
        }
    
        sub cleanup() {
           for ( @_ ) {
              s/s+/ /g;
           }
        }
    
        sub workDir() {
        # Use home directory to process data
        chdir or die "$!";
        if ( ! -d $processdir ) {
           mkdir ("$ENV{HOME}/$processdir", 0755) or die "Cannot make directory $processdir: $!";
           }
        }  



But as this-above script-unfortunatley does not take care for the separators i have had to take care for a method, that does look for separators. In order to get the data (output) separated. 

So with the separation i am able to work with the data - and store it in a mysql-table.. or do something else...So here [below] are the bits - that work out the csv-formate Note - i want to put the code below into the code above - to combine the spider-logic of the above mentioned code with the logic of outputting the data in CSV-formate.
where to set in the code Question: can we identify this point to migrate the one into the other... !? 
That would be amazing... I hope i could make clear what i have in mind...!? Are we able to use the benefits of the both parts (/scripts ) migrating them into one? 

So the question is: where to set in with the CSV-Script into the script (above)


    #!/usr/bin/perl
    use warnings;
    use strict;
    use LWP::Simple;
    use HTML::TableExtract;
    use Text::CSV;
    
    my $html= get 'http://192.68.214.70/km/asps/schulsuche.asp?q=a&a=20';
    $html =~ tr/\r//d;     # strip carriage returns
    $html =~ s/&nbsp;/ /g; # expand spaces
    
    my $te = new HTML::TableExtract();
    $te->parse($html);
    
    my @cols = qw(
        rownum
        number
        name
        phone
        type
        website
    );
    
    my @fields = qw(
        rownum
        number
        name
        street
        postal
        town
        phone
        fax
        type
        website
    );
    
    my $csv = Text::CSV->new({ binary => 1 });
    
    foreach my $ts ($te->table_states) {
        foreach my $row ($ts->rows) {
    
            #  trim leading/trailing whitespace from base fields
            s/^\s+//, s/\s+$// for @$row;
    
            # load the fields into the hash using a "hash slice"
            my %h;
            @h{@cols} = @$row;
    
            # derive some fields from base fields, again using a hash slice
            @h{qw/name street postal town/} = split /\n+/, $h{name};
            @h{qw/phone fax/} = split /\n+/, $h{phone};
    
            #  trim leading/trailing whitespace from derived fields
            s/^\s+//, s/\s+$// for @h{qw/name street postal town/};
    
            $csv->combine(@h{@fields});
            print $csv->string, "\n";
        }
    }



The thing is that i have had very good results with the first script! It fetches the data from the page: http://192.68.214.70/km/asps/schulsuche.asp?q=n&a=20
(note 6142 records) - But note - the data are not separated...!

And i have a second script. This part can do the CSV-formate. i want to combine it with the spider-logic.

where is the part to insert?  I look forward to any and all help.

if i have to be more precice - just let me know... 


zero



How to make LWP and HTML::TableExtract spitting out CSV with Text::CSV
===

first of all: sorry for being the novice. I am just beginning with Perl. So  don´t be rude with me if i ask beginner-questions. 

This all is a great learning curve for me.

first of all - this is a great community and a very very nice place to be! I love this great place. Today, one last question regarding a parser... I am currently working on a  little parser.

i have had very good results with the first script! This was able to run great! 
It fetches the data from the page: http://192.68.214.70/km/asps/schulsuche.asp?q=n&a=20
(note 6142 records) - But note - the data are not separated, so the subequent work with the data is a bit difficult. 

it spits out the data like this: 

    lfd. Nr. Schul- nummer Schulname Straße PLZ Ort Telefon Fax Schulart Webseite
    1 0401 Mädchenrealschule Marienburg, Abenberg, der Diözese Eichstätt Marienburg 1 91183  Abenberg   09178/509210  Realschulen  mrs-marienburg.homepage.t-online.de 
    2 6581 Volksschule Abenberg (Grundschule) Güssübelstr. 2 91183  Abenberg   09178/215 09178/905060 Volksschulen  home.t-online.de/home/vs-abenberg 
    3 6913 Mittelschule Abenberg  Güssübelstr. 2 91183  Abenberg   09178/215 09178/905060 Volksschulen  home.t-online.de/home/vs-abenberg 
    4 0402 Johann-Turmair-Realschule Staatliche Realschule Abensberg Stadionstraße 46 93326  Abensberg   09443/9143-0,12,13 09443/914330 Realschulen  www.rs-abensberg.de 

But i need to separate the data: with Commas or someting like that! 

And i have a second script. This part can do the CSV-formate. i want to ombine it with the spider-logic. But first lets have a look at the first script: with the great spider-logic. 

see the code that is appropiate: 

     #!/usr/bin/perl
        use strict;
        use warnings;
        use HTML::TableExtract;
        use LWP::Simple;
        use Cwd;
        use POSIX qw(strftime);
        my $te = HTML::TableExtract->new;
        my $total_records = 0;
        my $suchbegriffe = "e";
        my $treffer = 50;
        my $range = 0;
        my $url_to_process = "http://192.68.214.70/km/asps/schulsuche.asp?q=";
        my $processdir = "processing";
        my $counter = 50;
        my $displaydate = "";
        my $percent = 0;
    
        &workDir();
        chdir $processdir;
        &processURL();
        print "\nPress <enter> to continue\n";
        <>;
        $displaydate = strftime('%Y%m%d%H%M%S', localtime);
        open OUTFILE, ">webdata_for_$suchbegriffe\_$displaydate.txt";
        &processData();
        close OUTFILE;
        print "Finished processing $total_records records...\n";
        print "Processed data saved to $ENV{HOME}/$processdir/webdata_for_$suchbegriffe\_$displaydate.txt\n";
        unlink 'processing.html';
        die "\n";
    
        sub processURL() {
        print "\nProcessing $url_to_process$suchbegriffe&a=$treffer&s=$range\n";
        getstore("$url_to_process$suchbegriffe&a=$treffer&s=$range", 'tempfile.html') or die 'Unable to get page';
    
           while( <tempfile.html> ) {
              open( FH, "$_" ) or die;
              while( <FH> ) {
                 if( $_ =~ /^.*?(Treffer <b>)(d+)( - )(d+)(</b> w+ w+ <b>)(d+).*/ ) {
                    $total_records = $6;
                    print "Total records to process is $total_records\n";
                    }
                 }
                 close FH;
           }
           unlink 'tempfile.html';
        }
    
        sub processData() {
           while ( $range <= $total_records) {
              getstore("$url_to_process$suchbegriffe&a=$treffer&s=$range", 'processing.html') or die 'Unable to get page';
              $te->parse_file('processing.html');
              my ($table) = $te->tables;
              for my $row ( $table->rows ) {
                 cleanup(@$row);
                 print OUTFILE "@$row\n";
              }
              $| = 1; 
              print "Processed records $range to $counter";
              print "\r";
              $counter = $counter + 50;
              $range = $range + 50;
              $te = HTML::TableExtract->new;
           }
        }
    
        sub cleanup() {
           for ( @_ ) {
              s/s+/ /g;
           }
        }
    
        sub workDir() {
        # Use home directory to process data
        chdir or die "$!";
        if ( ! -d $processdir ) {
           mkdir ("$ENV{HOME}/$processdir", 0755) or die "Cannot make directory $processdir: $!";
           }
        }  



But as this-above script-unfortunatley does not take care for the separators i have had to take care for a method, that does look for separators. In order to get the data (output) separated. 

So with the separation i am able to work with the data - and store it in a mysql-table.. or do something else...So here [below] are the bits - that work out the csv-formate Note - i want to put the code below into the code above - to combine the spider-logic of the above mentioned code with the logic of outputting the data in CSV-formate.
where to set in the code Question: can we identify this point to migrate the one into the other... !? 
That would be amazing... I hope i could make clear what i have in mind...!? Are we able to use the benefits of the both parts (/scripts ) migrating them into one? 

So the question is: where to set in with the CSV-Script into the script (above)


    #!/usr/bin/perl
    use warnings;
    use strict;
    use LWP::Simple;
    use HTML::TableExtract;
    use Text::CSV;
    
    my $html= get 'http://192.68.214.70/km/asps/schulsuche.asp?q=a&a=20';
    $html =~ tr/\r//d;     # strip carriage returns
    $html =~ s/&nbsp;/ /g; # expand spaces
    
    my $te = new HTML::TableExtract();
    $te->parse($html);
    
    my @cols = qw(
        rownum
        number
        name
        phone
        type
        website
    );
    
    my @fields = qw(
        rownum
        number
        name
        street
        postal
        town
        phone
        fax
        type
        website
    );
    
    my $csv = Text::CSV->new({ binary => 1 });
    
    foreach my $ts ($te->table_states) {
        foreach my $row ($ts->rows) {
    
            #  trim leading/trailing whitespace from base fields
            s/^\s+//, s/\s+$// for @$row;
    
            # load the fields into the hash using a "hash slice"
            my %h;
            @h{@cols} = @$row;
    
            # derive some fields from base fields, again using a hash slice
            @h{qw/name street postal town/} = split /\n+/, $h{name};
            @h{qw/phone fax/} = split /\n+/, $h{phone};
    
            #  trim leading/trailing whitespace from derived fields
            s/^\s+//, s/\s+$// for @h{qw/name street postal town/};
    
            $csv->combine(@h{@fields});
            print $csv->string, "\n";
        }
    }



The thing is that i have had very good results with the first script! It fetches the data from the page: http://192.68.214.70/km/asps/schulsuche.asp?q=n&a=20
(note 6142 records) - But note - the data are not separated...!

And i have a second script. This part can do the CSV-formate. i want to combine it with the spider-logic.

where is the part to insert?  I look forward to any and all help.

if i have to be more precice - just let me know... 


zero


