I have the some 50 raw HTML page contents which are relevant to my project. I am not sure these contents are having unique pattern.
I need to parse the contents from all pages and has to be classified based on the keywords.
Keywords all like that
'REVIEWS',"REPORTS","FEEDBACK","DESCRIPTION","COMMENTS","SUCCESS RATES","FAILURE RATES"
The crawled HTML content has to be classified and mapped to the relevant keywords.
Also need to be split the contents and it's headers from the page for comparison
I am using Python.
Would you please suggest the way to do this? Which will be suitable to choose? How the idea has to be organised?
If you need to do classification given the content of pages, I would suggest you to take a look at NLTK (http://www.nltk.org/), a natural language toolkit of open source python modules.
Don't just try to look at occurrences of e.g. "report" in the the pages. A report may or may not have "report" as a title or in the content. You can use NLTK to find terms related to your keywords (e.g. success rates vs. approval rates), or from the same family (e.g. description vs. described).
Take a look at the pages' contents and try to define what sets them apart from the others. For instance, a page with comments will probably have expressions such as "I think that", "in my opinion" and subjective terms, usually adjectives and adverbs, like "good", "quickly", "horrible", etc. A report is unlikely to have such words in it.
Apart from the content, the structure of the page may vary from category to category. If you intend to analyse that, maybe using Beautiful Soup (http://www.crummy.com/software/BeautifulSoup/) for parsing is a good idea.
Parsing and splitting multiple HTML pages without having a clue