How to index tags in a string to ignore them in an NLP pipeline
I believe this case is generic and not only an ML case. But I also think this is very common dealing with languages (I imagine, as I went through this before).
Having a string like:
hello world <tag>blablablabla hellooooo</tag> and so on
These are user inputs so I would receive any possible HTML. Then I run text through a pipeline of multiple operations:
* Check for diss words
* Mask any privacy text like emails, phones, credit cards, etc.
* Removes common confusables
* etcetera
Having these operations set, some are external libraries and I don't want to change anything. All of these are language processing (NLP) functions so I also assume html tags might lower their efficiency or completely confuse them.
So what I want to do is to pass the String stripped from HTML, then reconstruct a string with a structure similar to the first input. For instance:
hello world <tag>blablablabla hellooooo</tag> and so on would be first
hello world blablabla hello and so on and finally
hello world <tag>blablabla hello</tag> and so on
Last thing to note, I have already one library that could strip HTML tags, and I think (waiting for response from maintainer) that it doesn't return tags indexes out of the box. 373
Assuming I could get tags indexes: One naive solution would be to keep HTML tags indexes but then it is not the best solution as strings might not only change but grow or shrink in size. But if this assumption would make it harder to implement an indexer then I think I would try to make it strict that the stripped HTML and the output of the pipeline would have the same length (somehow).
So I think this is very common (?), how do you deal with this ?
Edit: My only easy hack solution for this: is to replace all tags with whitespaces or asterisks, hopefully this is the least risky way assuming NLP libraries would:
1. Keep whitespaces or asterisks
2. Does not hurt their efficiency (whitespaces or asterisks... ignored ie no meaning) 373