In our other videos we talked about the basics of fine-tuning a language model with Tensorflow (and as always, when I refer to videos I'll link them below). Still, can we do better? So here's the code from our model fine-tuning video, and while it works, we could definitely tweak a couple of things. By far the most important thing is the learning rate. In this video we'll talk about how to change it, which will make your training much more consistently successful. In fact, there are two things we want to change about the default learning rate for Adam. The first is that it's way too high for our models - by default Adam uses a learning rate of 10^-3 1 e minus 3, which is very high for training Transformers. We're going to start at 5 by 10^-5 5 e minus 5, which is 20 times lower than the default. And secondly, we don't just want a constant learning rate - we can get even better performance if we 'decay' the learning rate down to a tiny value, or even 0, over the course of training. That's what this PolynomialDecay schedule thing is doing. That name might be intimidating, especially if you only vaguely remember what a polynomial is from maths class. However, all we need to do is tell it how long training is going to be, so it decays at the right speed - that's what this code here is doing. We're computing how many minibatches the model is going to see over its entire training run, which is the size of the training set, divided by the batch_size to get the number of batches per epoch, and then multiplied by the number of epochs to get the total number of batches across the whole training run. Once we know how many training steps we're taking, we just pass all that information to the scheduler and we're ready to go. What does the polynomial decay schedule look like? With default options, it's actually just a linear schedule, so it looks like this - it starts at 5e-5, which means 5 times ten to the minus 5, and then decays down at a constant rate until it hits zero right at the very end of training. So why do they call it polynomial and not linear? Because if you tweak the options, you can get a higher-order decay schedule, but there's no need to do that right now. Now, how do we use our learning rate schedule? Easy, we just pass it to Adam! You'll notice the first time when we compiled the model, we just passed it the string "adam". Keras recognizes the names of common optimizers and loss functions if you pass them as strings, so it saves time to do that if you only want the default settings. But we're professional machine learners now, with our very own learning rate schedule, so we have to do things properly. So first we import the optimizer, then we initialize it with our scheduler, and then we compile the model using the new optimizer, and whatever loss function you want - this will be sparse categorical crossentropy if you're following along from the fine-tuning video. And now we have a high-performance model, ready to go. All that remains is to fit the model just like we did before! Remember, because we compiled the model with the new optimizer with the new learning rate schedule, we don't need to change anything here. We just call fit again, with exactly the same command as before, but now we get beautiful training with a nice, smooth learning rate decay.