  There is a perceived trade-off between machine learning code that is easy to
write, and machine learning code that is scalable or fast to execute. In
machine learning, imperative style libraries like Autograd and PyTorch are easy
to write, but suffer from high interpretive overhead and are not easily
deployable in production or mobile settings. Graph-based libraries like
TensorFlow and Theano benefit from whole-program optimization and can be
deployed broadly, but make expressing complex models more cumbersome. We
describe how the use of staged programming in Python, via source code
transformation, offers a midpoint between these two library design patterns,
capturing the benefits of both. A key insight is to delay all type-dependent
decisions until runtime, via dynamic dispatch. We instantiate these principles
in AutoGraph, a software system that improves the programming experience of the
TensorFlow library, and demonstrate usability improvements with no loss in
performance compared to native TensorFlow graphs. We also show that our system
is backend agnostic, and demonstrate targeting an alternate IR with
characteristics not found in TensorFlow graphs.

Programming Languages;Machine Learning;Machine Learning