Text Embedding Models Contain Bias
It&#x27;s hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the sake of example) a psychological evaluation application. Now imagine that after years of use you discover that your model systematically marks african americans as less psychologically fit than white americans. Who would be to blame? Naturally, you would. Your actions led to a biased model being used to unjustly and arbitrarily harm innocent people, and your leadership would be right to call into question every decision your application ever made.<p>Now imagine the same scenario except your app was trained on data instead of hand-written. Make no mistake, the answer to the question of who&#x27;s to blame is exactly the same: the developer. The response should be exactly the same: a complete loss of confidence in the model.<p>I&#x27;m appalled that this needs to be said, but reading this comments section I&#x27;m afraid it does: <i>Machine learning models are inference and pattern recognition devices, not scientific tools</i>. They don&#x27;t magically reveal hidden patterns in the world, they repeat the patterns that the developers train them on. If you trained a machine learning model to perform psychological evaluations [1] or sentence convicts [2] or recognize faces [3], and your model is biased in a way that is unnecessary and unjust, your model is bad you should be held accountable for its failures.<p>[1] <a href="https:&#x2F;&#x2F;affect.media.mit.edu&#x2F;projects.php?id=4079" rel="nofollow">https:&#x2F;&#x2F;affect.media.mit.edu&#x2F;projects.php?id=4079</a><p>[2] <a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;2017&#x2F;05&#x2F;01&#x2F;us&#x2F;politics&#x2F;sent-to-prison-by-a-software-programs-secret-algorithms.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;2017&#x2F;05&#x2F;01&#x2F;us&#x2F;politics&#x2F;sent-to-priso...</a><p>[3] <a href="https:&#x2F;&#x2F;www.wnycstudios.org&#x2F;story&#x2F;deep-problem-deep-learning&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.wnycstudios.org&#x2F;story&#x2F;deep-problem-deep-learning...</a>