NLP's Clever Hans Moment Has Arrived
Great article, and some great links from it too. The list of models "following the letter but not the spirit" at <a href="https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml" rel="nofollow">https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...</a> has some hilarious and creative examples that one would hope never make it into production:<p>> AI trained to classify skin lesions as potentially cancerous learns that lesions photographed next to a ruler are more likely to be malignant.<p>> Agent pauses the game indefinitely to avoid losing<p>> A robotic arm trained to slide a block to a target position on a table achieves the goal by moving the table itself.<p>> Evolved player makes invalid moves far away in the board, causing opponent players to run out of memory and crash<p>> Genetic algorithm for image classification evolves timing attack to infer image labels based on hard drive storage location<p>> Deep learning model to detect pneumonia in chest x-rays works out which x-ray machine was used to take the picture; that, in turn, is predictive of whether the image contains signs of pneumonia, because certain x-ray machines (and hospital sites) are used for sicker patients.<p>> Creatures bred for speed grow really tall and generate high velocities by falling over<p>> Neural nets evolved to classify edible and poisonous mushrooms took advantage of the data being presented in alternating order, and didn't actually learn any features of the input images