Business as usual has focused primarily on steadily improving tools for function 
approximation and composition within the deep learning toolbox, and on gathering 
larger training sets and scaling to increasingly larger clusters of GPUs and TPUs. One 
can imagine improving a system like GPT-2 by gathering larger data sets, augmenting 
those data sets in various ways, and incorporating various kinds of improvements in 
the underlying architecture. While there is value in such approaches, a more 
fundamental rethink is required. 

Many more drastic approaches might be pursued. Yoshua Bengio, for example, has 
made a number of sophisticated suggestions for significantly broadening the toolkit of 
deep learning, including developing techniques for statistically extracting causal 
relationships through a sensitivity to distributional changes (Bengio et al., 2019) and 
techniques for automatically extracting modular structure (Goyal et al., 2019), both of 
which I am quite sympathetic to.
The Next Decade in AI-