< > changed to < and > while parsing html with beautifulsoup in python
It can be due to an invalid character (due to charset encoding/decoding), therefor BeautifulSoup has issues to parse the input.
I solve it by passing my string directly to BeautifulSoup without doing any encoding/decoding.
In my case, I was trying to convert UTF-16 to UTF-8 myself.
While processing html using Beautifulsoup, the < and > were converted to &lt;and &gt;, since the tag anchor were all converted, the whole soup lost its structure, any suggestion?