top-1 accuracy on a linear probe of our features.
 
 Summary of the approach. Taken from the original paper. 
This model was contributed by nielsr, based on this issue. The original code can be found
here.
Tips:
Demo notebooks for ImageGPT can be found
  here.
ImageGPT is almost exactly the same as GPT-2, with the exception that a different activation
  function is used (namely "quick gelu"), and the layer normalization layers don't mean center the inputs. ImageGPT
  also doesn't have tied input- and output embeddings.
As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence
  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
