How to design a HTML parser that would follow the Single Responsibility Principle?
I think you're interpreting the Single-Responsibility Principle (SRP) a little differently. It's actual meaning is a little off from 'a class should do only one thing'. It actually states that a class should have one and only one reason to change.
To employ the SRP you have to ask yourself to what/who would your parser module methods be responsible, what/who might make them change. If the answer for each method is the same, then your Parser class employs the SRP correctly. If there are methods that are responsible to different things (business-rule givers, groups of users etc.) then those methods should be taken out and be placed elsewhere.
Your overall objective with the SRP is to protect your class from changes coming from different directions.
I am writing an application which extracts some data from HTML using BeautifoulSoup4. These are search results of some kind, to be more specific. I thought it would be a good a idea to have a Parser class, storing default values like URL prefixes, request headers etc. After configuring those parameters, the public method would return a list of objects, each of them containing a single result or maybe even an object with a list composed into it alongside with some other parameters. I'm struggling to decouple small pieces of logic that build that parser implementation from the parser class itself. I want to write dozens of parser private utility methods like: _is_next_page_available, _are_there_any_results, _is_did_you_mean_available etc. However, these are the perfect candidates for writing unit tests! And since I want to make them private, I have a feeling that I'm missing something...
My other idea was to write that parser as a function, calling bunch of other utility functions, but that would be just equal to making all of those methods public, which doesn't make sense, since they're implementation details.
Could you please advice me how to design this properly?