Accelerated PyTorch Training on M1 Mac
This is really cool for a number of reasons:<p>1.) Apple Silicon <i>currently</i> can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now.<p>Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency<p>2.) Apple Silicon probably <i>will</i> compete directly with Nvidia GPUs in the near future in terms of raw compute power in future generations of products like the Mac Studio and Mac Pro, which is very exciting. Competition in this space is incredibly good for consumers.<p>3.) At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we'll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking <i>very</i> enticing.