NLP’s generalization problem, and how researchers are tackling it
To be fair, poor generalisation is not a problem of NLP, but of NLP using deep
neural networks, which is a very recent phenomenon, following from the success
of deep learning for image and speech processing.<p>If you asked me (you shouldn&#x27;t, I&#x27;m biased), I&#x27;d tell you that we&#x27;re never going to get deep
neural nets to behave well enough to learn meaning. Those damn things are way
too smart for us. They&#x27;re so smart that they can always find the easiest,
dumbest way to map inputs to outputs- by simply overfitting to their training
set (or their entire data set, if cross-validation is thorough enough). You
can see lots of examples of that in the article above.<p>The big promise of deep learning was (and is) that it would free us from the
drudgery of feature engineering- but, the amount of work you need to do to
convince a deep net to learn what you want it to, rather than what it wants to
learn, is starting to approach that of hand-crafting features.<p>And we still don&#x27;t have language models that make any sense at all.