We're not Platonists, we've just learned the bitter lesson
This stuff drives me up the wall for a few reasons.<p>The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.<p>So how, in some detail, do we get from big-ass mixture model of transformers to even <i>self-operation</i>? That’s got to be a pre-requisite for <i>self-improvement</i> right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a <i>clear and present danger</i> of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes.<p>Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the <i>log scale and a ruler</i> methodology employed by Yud or whoever.<p>Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight.<p>But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it.<p>I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.