Some bugs in your code are very straightforward. You try running it, you get a syntax error somewhere, Python tells you exactly where, and you fix it. This is great - it's simple and satisfying. Sometimes, though, things crash and the error is impossible to understand. This happens a lot in machine learning for a few reasons - you're working with big data structures, using big, complex libraries with a lot of moving parts, and also you're doing a lot of GPU computing. In Keras there's the added bonus problem that your models are often compiled before execution, which is great for performance but makes debugging them very difficult. This is going to be a video about what to do when you run into one of those nightmare bugs. To give you some intuitions for what can go wrong, and where to look for the source of bugs that you encounter, let's use this example script, and I'll show it to you here in two parts. First, we do all our imports, we load a dataset, we create our tokenizer and we tokenize the dataset. Next, we convert our datasets to TensorFlow datasets, so that we can run fit() on them, and then we load our model from a pretrained checkpoint, compile it and fit it.  It seems straightforward enough, but beware! This spooky code hides many dark and mysterious secrets. What happens when we run it? Well, this isn't great. What does that mean? We tried to train on our data, but we got no gradient? This is pretty perplexing - how do we even begin to debug something like that? When the error you get doesn't immediately suggest where the problem is, the best solution is often to walk through things in sequence, making sure at each stage that things look right. And of course, the place to start is always to check your data. The best way to do that to grab a batch from the tf.data.Dataset that your model is training on, right at the end of the training pipeline. And we can do that like so, by looping over the dataset for one iteration and then breaking. So what do we get when we inspect that batch? We see that we're not getting any gradient because we're not passing labels to Keras! Our labels are in the batch, but they're a key in the input dictionary, not a separate label. This is one of the most common issues you'll encounter when training Transformers models with TensorFlow. Our models can all compute loss internally, but to use that loss for training the labels need to be passed in the input dictionary, where the model can see them. This internal loss is the loss that we use when we don't specify a loss value to compile(). Keras, on the other hand, usually expects labels to be passed separately from the input dictionary and not to be visible to the model, and loss computations will usually fail if you don't do that. We need to choose one or the other: Either we use the model's internal loss and keep the labels where they are, or we keep using Keras losses, but we move the labels to the place Keras expects them. For simplicity, let's use the model internal losses, by removing the loss argument from the call to compile(). So what happens if we try training with model.fit() after fixing the loss function! Well, it runs this time... but now we get a loss of nan. This isn't good. NaN is not a good loss. In fact, if we inspect our model now, we'll see that not only are all the outputs nan , all the weights are nan too. Once a single nan creeps into your computations, it tends to spread, because it propagates from the loss back through your gradient and then into the weight updates. So nan destroyed our model. But where did it creep in first? To find out, we need to re-initialize the model and look at the outputs for just the first batch. And when we do that, we see that nan first appears in the loss, but only in some samples! You can see this in more detail in the accompanying section of the course notes, but we find that if we look at the labels, the samples with a loss of nan all have a label of 2! This gives us a very strong clue - if we check the model, with model.config.num_labels, we see the model thinks there's only 2 labels, but if we see a value of 2, that means there's at least 3 labels, because 0 is a label too! So we got a loss of nan because we got an "impossible" label. To fix that, we need to go back and set the model to have the right number of labels. We can set num_labels=3 when we initialize the model with from_pretrained. So now we think our data is good and our model is good, so training should work. And if we try running model.fit(), we get... hmm. The loss goes down, but it's not very quick. And if we keep running it out, we'll find that it stalls at a fairly high value. What's going on? Well, when things are mostly working, but training is just slow, that can often be a good time to look at your optimizer and training hyperparameters. And this is where I want to mention one of the most common sources of issues when you're working with Keras - you can name things like optimizers with strings, but if you do that, all of the options get silently set to their default values. So we specified our optimizer as Adam, but in the process we invisibly got the default learning rate, which is 1e-3, or ten to the power of minus 3. This is way too high for training transformer models! We should go back and specify the learning rate directly - good values are between 1e-5 and 1e-4. Let's split the difference and pick 5e-5. And if you recompile with that, you'll find that training actually works, at last. Again, I went through this quite quickly, and I recommend checking out the course notes for this to see this in more detail and to experiment with the code yourself. Good luck, and remember to take breaks if your code is giving you a hard time!