Method for Generalising XPath selectors
From a table I am looking at on the web in firefox this is an xpath selector.

    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tbody/x:tr[2]/x:td[2]/x:a

So i remove /x:tbody because that was added by firefox. But how is it generalised to get a links in the table that have the same base Xpath. the only obvious difference is that tr increases by 1 for each link in the table.

    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tr[2]/x:td[2]/x:a
    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tr[3]/x:td[2]/x:a

If there are successive tables of links on the page. and the only difference to me appears that div increases from 1 to 2.

So second table link.

    id('ls-page')/x:div[5]/x:div[2]/x:div[2]/x:table/x:tr[2]/x:td[2]/x:a

    /x:div[5]/x:div[1]

becomes

    /x:div[5]/x:div[2]

1) Is there a method or process to use to generalize an XPATH selector?

2) For each table do i have to create two separate generalised functions one to retrieve tables and one to retrieve links from tables?

Note I am referring to this site [live nrl stats][1] . I have been reading scrapy documentation and beautifulsoup documentation but am open to any suggestions regarding tooling as I am just learning.


  [1]: http://live.nrlstats.com/nrl/season2012.html
Method for Generalising XPath selectors
===

From a table I am looking at on the web in firefox this is an xpath selector.

    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tbody/x:tr[2]/x:td[2]/x:a

So i remove /x:tbody because that was added by firefox. But how is it generalised to get a links in the table that have the same base Xpath. the only obvious difference is that tr increases by 1 for each link in the table.

    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tr[2]/x:td[2]/x:a
    id('ls-page')/x:div[5]/x:div[1]/x:div[2]/x:table/x:tr[3]/x:td[2]/x:a

If there are successive tables of links on the page. and the only difference to me appears that div increases from 1 to 2.

So second table link.

    id('ls-page')/x:div[5]/x:div[2]/x:div[2]/x:table/x:tr[2]/x:td[2]/x:a

    /x:div[5]/x:div[1]

becomes

    /x:div[5]/x:div[2]

1) Is there a method or process to use to generalize an XPATH selector?

2) For each table do i have to create two separate generalised functions one to retrieve tables and one to retrieve links from tables?

Note I am referring to this site [live nrl stats][1] . I have been reading scrapy documentation and beautifulsoup documentation but am open to any suggestions regarding tooling as I am just learning.


  [1]: http://live.nrlstats.com/nrl/season2012.html