EU regulations on algorithmic decision-making and a “right to explanation”
Disclaimer - I work in the financial services industry.<p>This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector.<p>This is really needed for a number of different reasons. The biggest is that algorithms are increasingly running our world. Even outside of mortgage, have you ever had to fork over the right to pull your credit score for either rent or (in some limited cases) to apply for a job? Not being able to understand:
a) Why you were rejected.
b) Why you are paying a higher interest rate. 
c) What you can do to fix that?<p>Housing and employment is fundamentally a question of human rights, not just banking profitability or academic algorithm improvements.<p>Part of the reason why Credit Scoring (which is intrinsically algorithmic decision-making) is because it replaced the &quot;old boys&quot; network that used to dominate. In this world you went to your banker, and the decision to extend credit was dominated by your personal relationship with a banker who might not be the same ethnicity, religion, nationality, etc as you would dominate your ability to purchase housing. The credit score democratized financial access.<p>It can still be used to discriminate. There have been a number of studies that have shown that simply being from a white community tends to increase the number of other people you can lean on in a emergency, and hence make you less likely to default on a loan. From a pure predictability point of view, a bank is at a lower risk of default lending to someone from that community, but that in turn denies financial access to people not in that community, continuing the trend.<p>A big problem in this kind of financial model is that it&#x27;s relatively easy to accidentally or deliberately find other variables that mirror &quot;protected&quot; vectors. A simplistic example is zip code, where the zip code reflects a area that is predominately ethnic minorities.<p>So it&#x27;s not cut and dried. It&#x27;s my PoV that It&#x27;s not predictability versus red-tape, and the people trying to do  unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism.