NLP's Clever Hans Moment Has Arrived
Great article, and some great links from it too. The list of models "following the letter but not the spirit" at https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... has some hilarious and creative examples that one would hope never make it into production:> AI trained to classify skin lesions as potentially cancerous learns that lesions photographed next to a ruler are more likely to be malignant.> Agent pauses the game indefinitely to avoid losing> A robotic arm trained to slide a block to a target position on a table achieves the goal by moving the table itself.> Evolved player makes invalid moves far away in the board, causing opponent players to run out of memory and crash> Genetic algorithm for image classification evolves timing attack to infer image labels based on hard drive storage location> Deep learning model to detect pneumonia in chest x-rays works out which x-ray machine was used to take the picture; that, in turn, is predictive of whether the image contains signs of pneumonia, because certain x-ray machines (and hospital sites) are used for sicker patients.> Creatures bred for speed grow really tall and generate high velocities by falling over> Neural nets evolved to classify edible and poisonous mushrooms took advantage of the data being presented in alternating order, and didn't actually learn any features of the input images