Update

ErnWZxJovaM.json

@@ -0,0 +1,468 @@
+[
+    {
+        "num_chapter": 0,
+        "title": "Introduction to the Course and AI Advancements",
+        "start_paragraph_number": 0,
+        "end_paragraph_number": 17,
+        "start_time": 8,
+        "end_time": 467,
+        "paragraphs": [
+            "Good afternoon, everyone, and welcome to MIT 6.S191. My name is Alexander Amini, and I'll be one of your instructors for the course this year, along with Ava. We're really excited to welcome you to this incredible course.",
+            "This is a fast-paced and intense one-week course that we're about to go through together. We'll be covering the foundations of a rapidly changing field, and a field that has been revolutionizing many areas of science, mathematics, physics, and more.",
+            "Over the past decade, AI and deep learning have been rapidly advancing and solving problems that we didn't think were solvable in our lifetimes. Today, AI is solving problems beyond human performance, and each year, this lecture is getting harder and harder to teach because it's supposed to cover the foundations of the field.",
+            "If you think about any other introductory course, like a 101 course on mathematics or biology, the first lecture doesn't really change much over time. But we're in a rapidly changing field of AI and deep learning where even these types of lectures are rapidly changing.",
+            "Let me give you an example of how we introduced this course only a few years ago: \"Hi, everybody, and welcome to MIT 6.S191, the official introductory course on deep learning, taught here at MIT. Deep learning is revolutionizing many fields, from robotics to medicine and everything in between. You'll learn the fundamentals of this field and how you can build these incredible algorithms.\"",
+            "In fact, this entire speech and video are not real and were created using deep learning and artificial intelligence.",
+            "I'm excited to share my knowledge with you as we explore the world of artificial intelligence and deep learning together. We'll be covering a lot of ground in this course, and I'm looking forward to it.",
+            "Artificial intelligence and deep learning have come a long way in this class. It's been an honor to speak with you today, and I hope you enjoy the course.",
+            "The really surprising thing about that video to me was how viral it went a few years ago. When we first did it, it was shocking to see how quickly it spread. Within a couple of months, the video got over a million views. People were amazed by the realism of AI, which could generate content that looked and sounded extremely hyperrealistic.",
+            "When we created that video for the class a few years ago, it took us about $10,000 and a lot of compute to generate just about a minute-long video. That's extremely expensive to compute something that looks like that. Maybe some of you aren't even impressed by the technology today because you see all the amazing things that AI and deep learning are producing now.",
+            "Fast-forward to today, and the progress in deep learning is incredible. People were making all kinds of exciting remarks about it when it came out a few years ago. Now, this is common stuff because AI is really doing much more powerful things than this fun little introductory video.",
+            "So, today, fast-forward four years, and we're in a completely different place. AI is now generating content with deep learning being so commoditized. Deep learning is in all of our fingertips now, online in our smartphones, and so on. In fact, we can use deep learning to generate these types of hyperrealistic pieces of media and content entirely from English language without even coding anymore.",
+            "Before, we had to actually go in and train these models and really code them to be able to create that one-minute-long video. Today, we have models that will do that for us end-to-end directly from English language. We can use these models to create something that the world has never seen before.",
+            "Here's the improved version of the text:",
+            "'Imagine a photo of an astronaut riding a horse. These models can create those pieces of content entirely from scratch. My personal favorite is actually how we can now ask these deep learning models to create new types of software, even software that can ask them to create something. For example, we can ask a neural network to write TensorFlow code to train another neural network. Our model can produce examples of functional and usable pieces of code that satisfy this English prompt, while walking through each part of the code independently, so not just producing it but actually educating and teaching the user on what each part of these code blocks are actually doing.",
+            "You can see examples here. What I'm trying to show you with all of this is that this is just highlighting how far deep learning has gone, even in just a couple of years since we've started teaching this course. Going back even from before that, to eight years ago, the most amazing thing that you'll see in this course, in my opinion, is that what we try to do here is to teach you the foundations of all of this, how all of these different types of models are created from the ground up, and how we can make all of these amazing advances possible, so that you can also do it on your own as well.",
+            "As I mentioned at the beginning, this introduction course is becoming increasingly challenging every year, and I'm unsure where the field will be headed next year. Honestly, it's difficult to predict, given how rapidly it's evolving. However, I do know that what we'll share with you in this course, over the next week, will provide the foundations of all the tech technologies we've seen so far, enabling you to create your own future and design brand new deep learning models using those fundamentals."
+        ],
+        "paragraph_timestamps": [
+            8,
+            21,
+            65,
+            94,
+            113,
+            144,
+            457,
+            160,
+            174,
+            215,
+            236,
+            253,
+            284,
+            300,
+            302,
+            353,
+            386
+        ]
+    },
+    {
+        "num_chapter": 1,
+        "title": "Foundations of Intelligence and Deep Learning",
+        "start_paragraph_number": 17,
+        "end_paragraph_number": 24,
+        "start_time": 467,
+        "end_time": 567,
+        "paragraphs": [
+            "Let's get started by exploring the foundations from the very beginning and asking ourselves what intelligence is at its core \u2013 not just artificial intelligence, but intelligence itself. In my opinion, intelligence is the ability to process information, which informs our future decision-making abilities. This is something we humans do every day.",
+            "Artificial intelligence is simply the ability to give computers the same capacity to process information and inform future decisions. Machine learning is a subset of artificial intelligence, and it's the science of teaching computers how to process information and make decisions from data, rather than hardcoding rules into machines.",
+            "Deep learning is a subset of machine learning that uses neural networks to process raw, unprocessed data and ingest large datasets.",
+            "'Let's start by tackling the foundations from the beginning and exploring what intelligence is at its core. We'll examine how this concept of deep learning relates to other scientific concepts you've learned about so far.'",
+            "'Now, let's break down the key components: artificial intelligence, machine learning, and deep learning. Artificial intelligence is the ability to give computers the capacity to process information and inform future decisions. Machine learning is the science of teaching computers how to process information and make decisions from data. Deep learning is a subset of machine learning that uses neural networks to process raw data and ingest large datasets.'",
+            "'I'd like to emphasize that intelligence is the ability to process information, which informs our future decision-making abilities. This is something we humans do every day. Artificial intelligence is simply the ability to give computers the same capacity to process information and inform future decisions.'",
+            "Now, let's dive deeper into the concept of deep learning and how it relates to other scientific concepts you've learned about so far."
+        ],
+        "paragraph_timestamps": [
+            467,
+            487,
+            537,
+            457,
+            487,
+            476,
+            558
+        ]
+    },
+    {
+        "num_chapter": 2,
+        "title": "Course Overview and Structure",
+        "start_paragraph_number": 24,
+        "end_paragraph_number": 37,
+        "start_time": 567,
+        "end_time": 833,
+        "paragraphs": [
+            "That's exactly what this class is all about: teaching machines how to process data, process information, and inform decision-making abilities from that data, and learn from that data. This program is split between two different parts, so you should think of this class as being comprised of both technical lectures and software labs.",
+            "We'll have several new updates this year, covering the rapid changing advances in AI, especially in some of the later lectures. The first lecture today will cover the foundations of neural networks themselves, starting with the building blocks of every single neural network, which is called the perceptron. We'll conclude with a series of exciting guest lectures from industry-leading sponsors of the course.",
+            "On the software side, after every lecture, you'll also get software experience and project-building experience to be able to take what we teach in lectures and actually deploy them in real code and produce something based on the learnings that you find in this lecture. At the very end of the class, from the software side, you'll have the ability to participate in a really fun day, which is the project pitch competition - it's kind of like a shark tank-style competition of all of the different projects from all of you, and win some really awesome prizes.",
+            "Here is the rewritten syllabus with improved readability:",
+            "The syllabus for this lecture is as follows:",
+            "Each day, we will have dedicated software labs that mirror the technical lectures, helping you reinforce your learnings. These labs are coupled with prizes for the top-performing software solutions.",
+            "The course will start with Lab One, which focuses on music generation. You will learn how to build a neural network that can learn from musical songs and then compose brand new songs in the same genre.",
+            "Tomorrow, Lab Two will cover computer vision, where you will learn about facial detection systems and build one from scratch using convolutional neural networks. You will also learn how to debias these systems, which is a significant problem in state-of-the-art solutions today.",
+            "Finally, a new Lab at the end of the course will focus on large language models. You will fine-tune a billion-parameter large language model to build an assistive chatbot and evaluate its cognitive abilities, including mathematics, scientific reasoning, and logical abilities.",
+            "At the end of the course, there will be a final project pitch competition, where each team will have up to 5 minutes to present their project. All of these labs are accompanied by great prizes, so there will be plenty of fun throughout the week.",
+            "There are many resources available to help with this class, which can be found on the course slides. If you have any questions, please post them on Piazza. The teaching team is also available to answer any questions you may have.",
+            "Myself and AA will be the main lecturers for this course, especially on Monday through Wednesday. We will also have some amazing guest lectures on the second half of the course, which cover the latest advancements in deep learning, particularly in industry outside of academia.",
+            "I'd like to take a moment to express my gratitude to all of our sponsors, without whom this course, like every year, would not be possible."
+        ],
+        "paragraph_timestamps": [
+            567,
+            598,
+            623,
+            661,
+            661,
+            664,
+            684,
+            698,
+            727,
+            748,
+            768,
+            781,
+            807
+        ]
+    },
+    {
+        "num_chapter": 3,
+        "title": "Introduction to Deep Learning",
+        "start_paragraph_number": 37,
+        "end_paragraph_number": 46,
+        "start_time": 833,
+        "end_time": 1044,
+        "paragraphs": [
+            "Now, let's dive into the technical aspects of the course, which is my favorite part. To understand why we care about deep learning, I think we need to go back a bit and understand how machine learning used to be performed.",
+            "Typically, machine learning would involve defining a set of features, which are essentially a set of things to look for in an image or piece of data. These features are usually hand-engineered by humans, but the problem with this approach is that they tend to be very brittle in practice.",
+            "The key idea of deep learning is to move away from hand-engineering features and rules, and instead, try to learn them directly from raw pieces of data. So, what are the patterns that we need to look at in data sets, such that if we look at those patterns, we can make some interesting decisions and take interesting actions?",
+            "For example, if we wanted to learn how to detect faces, we might look for certain patterns in a picture. What are we looking for to detect a face? We're looking for eyes, noses, and ears, and when those things are all composed in a certain way, we would probably deduce that it's a face. Computers do something very similar, so they have to understand what the patterns are that they look for, what the eyes and noses and ears of those pieces of data are, and then they can make decisions based on those patterns.",
+            "Computers do something very similar, so they have to understand what the patterns are that they look for, what the eyes and noses and ears of those pieces of data are, and then they can make decisions based on those patterns. They have to understand what the patterns are that they look for, what the eyes and noses and ears of those pieces of data are, and then they can make decisions based on those patterns.",
+            "The really interesting thing I think about deep learning is that these foundations for doing exactly what I just mentioned - picking out the building blocks, picking out the features from raw pieces of data and the underlying algorithms themselves - have existed for many, many decades now. The question I would ask at this point is, so why are we studying this now and why is all of this really blowing up right now and exploding with so many great advances?",
+            "Well, for one, there are three things. Number one is that the data that is available to us today is significantly more pervasive. These models are hungry for data, and you're going to learn about this more in detail, but these models are extremely hungry for data. And we're living in a world right now, quite frankly, where data is more abundant than it has ever been in our history.",
+            "Secondly, these algorithms are massively compute-hungry, and they're massively parallelizable, which means that they have greatly benefited from compute hardware, which is also capable of being parallelized. The particular name of that hardware is called a GPU. GPUs can run parallel processing streams of information and are particularly amenable to deep learning algorithms. And the abundance of GPUs and that compute hardware has also pushed forward what we can do in deep learning.",
+            "Finally, the last piece is the software. It's the open-source tools that are really used as the foundational building blocks of deploying and building all of these underlying models that you're going to learn about in this course. And those open-source tools have just become extremely streamlined, making this extremely easy for all of us to learn about these technologies within an amazing one-course like this."
+        ],
+        "paragraph_timestamps": [
+            833,
+            848,
+            871,
+            898,
+            911,
+            928,
+            958,
+            981,
+            1014
+        ]
+    },
+    {
+        "num_chapter": 4,
+        "title": "Introduction to Perceptrons",
+        "start_paragraph_number": 46,
+        "end_paragraph_number": 66,
+        "start_time": 1044,
+        "end_time": 1747,
+        "paragraphs": [
+            "So, let's start now with understanding exactly what is the fundamental building block of a neural network. That building block is called a perceptron. Every single perceptron, every single neural network is built up of multiple perceptrons, and you're going to learn how those perceptrons number one compute information themselves and how they connect to these much larger billion-parameter neural networks. The key idea of a perceptron, or even simpler, think of this as a single neuron. So, a neural network is composed of many, many neurons, and a perceptron is just one neuron. That idea of a perceptron is actually extremely simple, and I hope that by the end of today, this idea and this processing of a perceptron becomes extremely clear to you.",
+            "So, let's start by talking about just the forward propagation of information through a single neuron. Single neurons ingest information; they can actually ingest multiple pieces of information. So, here you can see this neuron taking in three pieces of information: X1, X2, and X(M). We define the set of inputs as x1 through M, and each of these inputs, each of these numbers, is going to be element-wise multiplied by a particular weight. So, this is going to be denoted here by W1 through WM, so this is a corresponding weight for every single input. You should think of this as really, you know, every weight being assigned to that input. The weights are part of the neuron itself.",
+            "Now, you multiply all of these inputs with their weights together and then you add them up. We take this single number after that addition and pass it through what's called a nonlinear activation function to produce your final output, which here, we'll be calling y.",
+            "Now, what I just said is not entirely correct, right? So, I missed out one critical piece of information. That piece of information is that we also have what you can see here, called the bias term. That bias term is actually what allows your neuron to shift its activation function horizontally on that x-axis, if you think of it. So, on the right side, you can now see this diagram illustrating that.",
+            "'Here, you can see the single equation that I talked through conceptually right now, mathematically written down as one single equation. We can actually rewrite this using linear algebra, using vectors and dot products. So, let's do that. Right now, our inputs are going to be described by a capital X, which is simply a vector of all of our inputs X1 through XM. And then our weights are going to be described by a capital W, which is going to be W1 through WM.'",
+            "'The input is obtained by taking the dot product of X and W. That dot product does an element-wise multiplication and then adds up all the element-wise multiplications. And then here's the missing piece: we're now going to add that bias term. We're calling the bias term W0, right? And then we're going to apply the nonlinearity, which here is denoted as Z or G, excuse me.'",
+            "'I've mentioned this nonlinearity a few times, this activation function. Let's dig into it a little bit more, so we can understand what is actually this activation function doing. Well, I said a couple of things about it. I said it's a nonlinear function, right? Here, you can see one example of an activation function, one common, uh, one commonly used activation function is called the sigmoid function, which you can actually see here on the bottom right-hand side of the screen.'",
+            "'The sigmoid function is very commonly used because it's outputs, right? It takes as input any real number, the x-axis is infinite, plus or minus, but on the Y-axis, it basically squashes every input X into a number between 0 and 1. So, it's actually a very common choice for things like probability distributions if you want to convert your answers into probabilities or learn or teach a neuron to learn a probability distribution.'",
+            "But in fact, there are many different types of nonlinear activation functions used in neural networks. Here are some common ones. Again, throughout this presentation, you'll see these little TensorFlow icons. These icons are used throughout the entire course to relate the foundational knowledge taught in lectures to the software labs. This might provide a good starting point for many of the pieces you'll need to do later on in the software parts of the class.",
+            "The sigmoid activation function, which we discussed in the last slide, is shown on the left-hand side. This is very popular because of its probability distributions. It squashes everything between zero and one. You'll also see two other very common types of activation functions in the middle and right-hand side: the tanh activation function and the relu activation function, also known as the rectified linear unit.",
+            "The relu activation function is very easy to compute and still has the nonlinearity we need. We'll talk about why we need it in a moment. It's very fast, just two linear functions piecewise combined with each other.",
+            "Now, let's talk about why we need a nonlinearity in the first place. Why not just deal with a linear function that we pass all of these inputs through? The point of the activation function is to introduce nonlinearity itself. What we want to do is allow our neural network to deal with nonlinear data. Our neural networks need the ability to deal with nonlinear data because the world is extremely nonlinear. This is important because, if you think of real-world datasets, this is just the way they are. If you look at datasets like this one, with green and red points, and I ask you to build a neural network to classify them, you'll see that the data is nonlinear. What can separate the green and the red points? This means that we actually need a nonlinear function to do that. We cannot solve this problem with a single line; in fact, if we used linear functions as our activation function, no matter how big our neural network is, it's still a linear function because linear functions combined with linear functions are still linear. So, no matter how deep or how many parameters your neural network has, the best it would be able to do to separate these green and red points would look like this.",
+            "But adding nonlinearity allows our neural networks to be smaller by allowing them to be more expressive and capture more complexities in the datasets, and this allows them to be much more powerful in the end.",
+            "So, let's understand this with a simple example. Imagine I give you a trained neural network. What does it mean, trained neural network? It means I'm giving you the weights, right, not only the inputs but I'm going to tell you what the weights of this neural network are. So, here, let's say the bias term w0 is going to be one, and our W vector is going to be 3 and ne2, right? These are just the weights of your trained neural network. Let's worry about how we got those weights in a second, but this network has two inputs, X1 and X2.",
+            "Now, if we want to get the output of this neural network, all we have to do is simply do the same story that we talked about before: it's a dot product, inputs with weights, add the bias, and apply the nonlinearity. Right, and those are the three components that you really have to remember as part of this class: dot product, add the bias, and apply a nonlinearity. That's going to be the process that keeps repeating over and over and over again for every single neuron after that happens, and that neuron is going to output a single number.",
+            "Let's take a look at what's inside that nonlinearity. It's simply a weighted combination of those inputs with those weights. So, what's inside G, right inside G, is a weighted combination of X and W, added with a bias, which is going to produce a single number. But in reality, for any input that this model could see, what this really is is a two-dimensional line because we have two parameters in this model. So, we can actually plot that line, and we can see exactly how this neuron separates points on these axes between X1 and X2. These are the two inputs of this model. We can see exactly and interpret exactly what this neuron is doing. We can visualize its entire space because we can plot the line that defines this neuron.",
+            "So, here we're plotting when that line equals zero. And in fact, if I give you that neuron a new data point \u2013 the new data point is X1 = -1 and X2 = 2, just an arbitrary point in this two-dimensional space \u2013 we can plot that point in the two-dimensional space. And depending on which side of the line it falls on, it tells us what the answer is going to be, what the sign of the answer is going to be, and also what the answer itself is going to be.",
+            "So, if we follow that equation written on the top here and plug in -1 and 2, we're going to get 1 - 3 - 4, which equals -6. Right, and when I put that into my nonlinearity G, I'm going to get a final output of 0.2. But don't worry about the final output \u2013 that's just going to be the output for that signal function. The important point to remember here is that the sigmoid function actually divides the space into these two parts. It squashes everything between Z and one, but it divides it implicitly by everything less than 0.5 and greater than 0.5, depending on if it's on the left side or the right side of the line.",
+            "So, depending on which side of the line you fall on \u2013 remember, the line is when X equals Z \u2013 the input to the sigmoid is zero. If you fall on the left side of the line, your output will be less than 0.5, because you're falling on the negative side of Z. The line, if your output is if your input is on the right side of the line, now your output is going to be greater than 0.5. Right, so here we can actually visualize this space \u2013 this is called the feature space of a neural network \u2013 we can visualize it in its entirety, right? We can totally visualize and interpret this neural network, and we can understand exactly what it's going to do for any input that it sees, right?",
+            "But of course, this is a very simple neuron, right? It's not a neural network; it's just one neuron, and even more than that, it's even a very simple neuron \u2013 it only has two inputs, right? So, in reality, the types of neurons that you're going to be dealing with in this course are going to be neurons and neural networks with millions or even billions of these parameters, right? So, here we only have two weights, W1 and W2, but today's neural networks have billions of these parameters. So, drawing these types of plots that you see here obviously becomes a lot more challenging \u2013 it's actually not possible."
+        ],
+        "paragraph_timestamps": [
+            1044,
+            1093,
+            1139,
+            1156,
+            1184,
+            1213,
+            1240,
+            1266,
+            1294,
+            1329,
+            1367,
+            1379,
+            1454,
+            1470,
+            1501,
+            1538,
+            1593,
+            1621,
+            1669,
+            1710
+        ]
+    },
+    {
+        "num_chapter": 5,
+        "title": "Building Neural Networks",
+        "start_paragraph_number": 66,
+        "end_paragraph_number": 71,
+        "start_time": 1747,
+        "end_time": 1874,
+        "paragraphs": [
+            "But now that we have some of the intuition behind a perceptron, let's start now by building neural networks and seeing how all of this comes together. So, let's revisit that previous diagram of a perceptron. Now, again, if there's only one thing to take away from this lecture, right now, it's to remember how a perceptron works \u2013 that equation of a perceptron is extremely important for every single class that comes after today, and there are only three steps: dot product with the inputs, add a bias, and apply your nonlinearity.",
+            "Let's simplify the diagram a little bit. I'll remove the weight labels from this picture, and now you can assume that if I show a line, every single line has an associated weight that comes with that line, right? I'll also remove the bias term for simplicity \u2013 assume that every neuron has that bias term; I don't need to show it.",
+            "Now, note that the result here, now calling it Z, which is just the dot product plus bias before the nonlinearity, is the output. The output is going to be linear \u2013 it's just a weighted sum of all those pieces we have, not applied the nonlinearity yet. But our final output is just going to be G of Z, it's the activation function or non-linear activation function applied to Z.",
+            "Now, if we want to step this up a little bit more and say what if we had a multi-output function? Now we don't just have one output, but let's say we want to have two outputs. Well, now we can just have two neurons in this network, right? Every neuron sees all of the inputs that came before it, but now you see the top neuron is going to be predicting an answer and the bottom neuron will predict its own answer.",
+            "Now, importantly, one thing you should really notice here is that each neuron has its own weights, right? Each neuron has its own lines that are coming into just that neuron, right? So they're acting independently, but they can later on communicate if you have another layer, right?"
+        ],
+        "paragraph_timestamps": [
+            1747,
+            1781,
+            1802,
+            1828,
+            1856
+        ]
+    },
+    {
+        "num_chapter": 6,
+        "title": "Building a Neural Network from Scratch",
+        "start_paragraph_number": 71,
+        "end_paragraph_number": 84,
+        "start_time": 1874,
+        "end_time": 2291,
+        "paragraphs": [
+            "So, let's start now by initializing this process a bit further and thinking about it more programmatically, right? What if we wanted to program this neural network ourselves from scratch, right? Remember that equation I told you \u2013 it didn't sound very complex. It's take a dot product, add a bias, which is a single number, and apply nonlinearity. Let's see how we would actually implement something like that.",
+            "So, to define the layer, we're going to call this a layer, which is a collection of neurons. We have to first define how that information propagates through the network, so we can do that by creating a call function here. First, we're going to actually define the weights for that network. So, remember, every network, every neuron \u2013 I should say, every neuron has weights and a bias. So, let's define those first.",
+            "We're going to create the call function to actually see how we can pass information through that layer. So, this is going to take in input and inputs, right? This is like what we previously called X, and it's the same. We're going to matrix multiply or take a dot product of our inputs with our weights, we're going to add a bias, and then we're going to apply a nonlinearity. It's really that simple. We've now created a single-layer neural network.",
+            "So, this line in particular, this is the part that allows us to be a powerful neural network, maintaining that nonlinearity. The important thing here is to note that modern deep learning toolboxes and libraries already implement a lot of these for you. So, it's important for you to understand the foundations, but in practice, all of that layer architecture and all that layer logic is actually implemented in tools like TensorFlow and PyTorch through a dense layer.",
+            "Here, you can see an example of calling or creating initializing a dense layer with two neurons, allowing it to feed in an arbitrary set of inputs. Here, we're seeing these two neurons in a layer being fed three inputs, right, and in code, it's only reduced down to this one line of TensorFlow code, making it extremely easy and convenient for us to use these functions and call them.",
+            "So, now let's look at our single-layered neural network. This is where we have now one layer between our input and our outputs, right? So, we're slowly and progressively increasing the complexity of our neural network, so that we can build up all of these building blocks. This layer in the middle is called a hidden layer, right, obviously because you don't directly observe it, you don't directly supervise it, right? You do observe the two input and output layers, but your hidden layer is just kind of a \u2013 a neuron layer that you don't directly observe, right? It just gives your network more capacity, more learning complexity.",
+            "Since we now have a transformation function from inputs to hidden layers and hidden layers to output, we now have a two-layered neural network, which means that we also have two weight matrices. We don't just have the W1 which we previously had to create this hidden layer, but now we also have W2, which does the transformation from the hidden layer to the output layer. Yes, what happens is nonlinearity in the hidden layer has just linear, so there's no perceptron or not. Yes, so every hidden layer also has a nonlinearity accompanied with it. That's a very important point because if you don't have that, it's just a very large linear function followed by a final nonlinearity at the very end. You need that cascading and overlapping application of nonlinearities that occur throughout the network. Awesome! Okay, so now let's zoom in and look at a single unit in the hidden layer. Take this one for example, let's call it Z2. It's the second neuron in the first layer. We compute its answer by taking a dot product of its weights with its inputs, adding a bias, and then applying a nonlinearity. If we took a different hidden node, like Z3, the one right below it, we would compute its answer exactly the same way that we computed Z2, except its weights would be different than the weights of Z2. Everything else stays exactly the same; it sees the same inputs.",
+            "Now, this picture is getting a little bit messy, so let's clean things up a little bit more. I'm going to remove all the lines and replace them with these boxes, these symbols that will denote what we call a fully connected layer. Right, so these layers now denote that everything in our input is connected to everything in our output, and the transformation is exactly as we saw before: dot product, bias, and nonlinearity.",
+            "And again, in code, to do this is extremely straightforward with the foundations that we've built up from the beginning of the class. We can now just define two of these dense layers: our hidden layer on line one with n hidden units, and then our output layer with two hidden output units.",
+            "Does that mean the nonlinearity function must be the same between layers? No, the nonlinearity function does not need to be the same through each layer. Often, it is because of convenience, but there are some cases where you would want it to be different as well, especially in lecture two, you're going to see nonlinearities be different even within the same layer, um, let alone different layers. But unless for a particular reason, generally, convention is there's no need to keep them differently.",
+            "Now, let's keep expanding our knowledge a little bit more. If we now want to make a deep neural network, not just a neural network like we saw in the previous slide, now it's deep. All that means is that we're now going to stack these layers on top of each other, one by one, more and more, creating a hierarchical model. Right, the ones where the final output is now going to be computed by going deeper and deeper and deeper into the neural network.",
+            "And again, doing this in code again follows the exact same story as before, just cascading these TensorFlow layers on top of each other and just going deeper into the network.",
+            "Okay, so now this is great because now we have at least a solid foundational understanding of how to not only define a single neuron but how to define an entire neural network, and you should be able to actually explain at this point or understand how information goes from input through an entire neural network to compute an output."
+        ],
+        "paragraph_timestamps": [
+            1874,
+            1899,
+            1924,
+            1964,
+            1996,
+            2024,
+            2063,
+            2159,
+            2182,
+            2201,
+            2232,
+            2259,
+            2270
+        ]
+    },
+    {
+        "num_chapter": 7,
+        "title": "Applying Neural Networks to Real Problems",
+        "start_paragraph_number": 84,
+        "end_paragraph_number": 90,
+        "start_time": 2291,
+        "end_time": 2451,
+        "paragraphs": [
+            "So now let's look at how we can apply these neural networks to solve a very real problem that, I'm sure, all of you care about. Here's a problem on how we want to build an AI system to learn to answer the following question: will I pass this class? I'm sure all of you are really worried about this question.",
+            "Um, so to do this, let's start with a simple input feature model. The feature\u2014the two features that let's concern ourselves with are going to be number one, how many lectures you attend, and\u2014",
+            "Number two: How many hours do you spend on your final project? So, let's look at some of the past years of this class. Right, we can actually observe how different people have lived in this space, right, between how many lectures and how much time you spend on your final project. And you can actually see every point is a person, the color of that point is going to be if they passed or failed the class, and you can see and visualize kind of this V-shaped feature space, if you will, that we talked about before.",
+            "Then, we have you, you fall right here, you're the point 45, uh, right in between the feature space. You've attended four lectures, and you will spend 5 hours on the final project. And you want to build a neural network to determine, given everyone else in the class, right, that I've seen from all of the previous years, you want to help you want to have your neural network help you to understand what is your likelihood that you will pass or fail this class.",
+            "So, let's do it. We now have all of the building blocks to solve this problem using a neural network. Let's do it. So, we have two inputs: those inputs are the number of lectures you attend and the number of hours you spend on your final project. It's four and five. We can pass those two inputs to our two variables, X1 and X2. These are fed into this single-layered, single-hidden-layered neural network, which has three hidden units in the middle. And we can see that the final predicted output probability for you to pass this class is 0.1, or 10%. Right, so very bleak outcome. It's not a good outcome. Um, the actual probability is one, right? So, attending four out of the five lectures and spending 5 hours in your final project, you actually lived in a part of the feature space which was actually very positive, right? It looked like you were going to pass the class.",
+            "So, what happened here? Anyone have any ideas? Why did the neural network get this so terribly wrong? Right, it's not trained exactly. So, this neural network is not trained. We haven't shown it any data yet, so it's just making random guesses."
+        ],
+        "paragraph_timestamps": [
+            2291,
+            2310,
+            2323,
+            2356,
+            2387,
+            2437
+        ]
+    },
+    {
+        "num_chapter": 8,
+        "title": "Introduction to Neural Networks",
+        "start_paragraph_number": 90,
+        "end_paragraph_number": 106,
+        "start_time": 2451,
+        "end_time": 2872,
+        "paragraphs": [
+            "Let's break down the concept of neural networks like this:",
+            "' Neural networks are like babies, right? Before they see any data, they haven't learned anything. There's no expectation that they should be able to solve any problems before we teach them something about the world. So, let's teach this neural network something about the problem first.",
+            "To train it, we first need to tell our neural network when it's making bad decisions. We need to teach it, really train it, to learn exactly like how we as humans learn. We have to inform the neural network when it gets the answer incorrect, so it can learn how to get the answer correct.",
+            "The closer the answer is to the ground truth, the smaller the loss should be, and the more accurate the model should be. For example, if the actual value for passing a class is 100%, but the neural network predicted a probability of 0.1, we compute what's called a loss.",
+            "Assuming we have data from many students who have taken the class before, we can plug all of them into the neural network and show them to the system. We care not only about how the neural network did on just one prediction but also about how it predicted on all of these different people it has shown in the past.",
+            "During the training and learning process, we want to find a network that minimizes the empirical loss between our predictions and those ground truth outputs, and we're going to do this on average across all of the different inputs the model has seen.",
+            "In the case of binary classification, where the output is a zero or one probability, we can use the sigmoid function to output a probability between 0 and 1. However, when it comes to multi-class classification or regression, we need to use a different function. The softmax function, also known as the softmax cross-entropy function, is used to output a probability distribution over multiple classes. This function is used to determine how well the neural network is performing, and the cross-entropy loss function is used to measure the difference between the predicted and true probability distributions.",
+            "In the case of regression, where we want to predict a real-valued output, we can use a different loss function. For example, we can use the mean squared error (MSE) loss function to measure the difference between the predicted and true values. This loss function is calculated by computing the difference between the predicted and true values, squaring it, and then taking the average.",
+            "Now, let's put all of this loss information together with the problem of training our neural network. We know that we want to find a neural network that will solve this problem on average, over all the data. This means that we're trying to find the optimal weights for our neural network, which we can represent as a large vector W.",
+            "To compute this Vector W, we need to find the optimal weights based on all of the data that we have seen so far. The vector W is also going to determine what is the loss, which is the deviation from the ground truth of our network, based on where it should be.",
+            "Remember that W is just a group of a bunch of numbers - a very big list of numbers, a list of weights for every single layer and every single neuron in our neural network. We want to find that Vector W based on a lot of data, which is the problem of training a neural network.",
+            "Our loss function is just a simple function of our weights. If we have only two weights in our neural network, like we saw earlier in the slide, then we can plot the loss landscape over this two-dimensional space. We have two weights W1 and W2, and for every single configuration or setting of those two weights, our loss will have a particular value, which is the height of this graph.",
+            "What we want to do is find the lowest point, where the loss is as good as possible. So, the smaller the loss, the better. We want to find the lowest point in this graph.",
+            "To do that, we start somewhere in this space, and we don't know where to start, so let's pick a random place to start. From that place, we compute what's called the gradient of the landscape at that particular point. This is a very local estimate of where the slope is increasing, basically where the slope is increasing at my current location.",
+            "That informs us not only where the slope is increasing, but more importantly, where the slope is decreasing. If I negate the direction, if I go in the opposite direction, I will move towards the lowest point in the graph. I can actually step down into the landscape and change my weights such that I lower my loss. Let's take a small step just a small step in the opposite direction of the part that's going up. Let's take a small step going down, and we'll keep repeating this process. We'll compute a new gradient at that new point, and then we'll take another small step, and we'll keep doing this over and over and over again until we converge at what's called a local minimum.",
+            "Right, so based on where we started, it may not be a global minimum of everywhere in this lost landscape, but let's find ourselves now in a local minimum and we're guaranteed to actually converge by following this very simple algorithm at a local minimum."
+        ],
+        "paragraph_timestamps": [
+            2451,
+            2453,
+            2470,
+            2495,
+            2519,
+            2539,
+            2560,
+            2625,
+            2675,
+            2703,
+            2727,
+            2749,
+            2788,
+            2798,
+            2823,
+            2857
+        ]
+    },
+    {
+        "num_chapter": 9,
+        "title": "Gradient Descent Algorithm",
+        "start_paragraph_number": 106,
+        "end_paragraph_number": 112,
+        "start_time": 2872,
+        "end_time": 3192,
+        "paragraphs": [
+            "So, let's summarize now this algorithm. This algorithm is called gradient descent. Let's summarize it first in pseudo code, and then we'll look at it in actual code in a second. So, there's a few steps. First, we initialize our location somewhere randomly in this weight space. Right, we compute the gradient of our loss at with respect to our weights. Okay, and then we take a small step in the opposite direction, and we keep repeating this in a loop over and over and over again. And we say we keep doing this until convergence, right, until we stop moving basically, and our Network basically finds where it's supposed to end up.",
+            "We'll discuss this small step, right? So, we're multiplying our gradient by a small step. We'll discuss that more in the later part of this lecture, but for now, let's also quickly show the analogous part in code as well, and it mirrors very nicely, right? So, we'll randomly initialize our weight, which happens every time you train a neural network. You have to randomly initialize the weights, and then you have a loop, right here showing it without even convergence, right? We're just going to keep looping forever, where we say, okay, we're going to compute the loss at that location, compute the gradient, update the weights, and then we'll do it again.",
+            "So, which way is up, and then we just negate that gradient, multiply it by some what's called the learning rate (LR), denoted here, it's a small step, and then we take a direction in that small step. Let's take a deeper look at this term here. This is called the gradient, right? This tells us which way is up in that landscape, and this again tells us even more than that. It tells us how our landscape, how our loss, is changing as a function of all of our weights. But I actually haven't told you how to compute this, so let's talk about that process. That process is called backpropagation. We'll go through this very briefly, and we'll start with the simplest neural network, uh, that's possible, right? So, we already saw the simplest building block, which is a single neuron. Now, let's build the simplest neural network, which is just a one-neuron neural network, right? So, it has one hidden neuron, it goes from input to hidden neuron to output, and we want to compute the gradient of our loss with respect to this weight W2, okay? So, I'm highlighting it here. We have two weights, let's compute the gradient first with respect to W2, and that tells us how much does a small change in W2 affect our loss? Does our loss go up or down if we move our W2 a little bit in one direction or another?",
+            "Let's write out this derivative. We can start by applying the chain rule backwards from the loss through the output, and specifically, we can decompose this derivative, or gradient, into two parts. The first part is DJ with respect to Dy, which is our output multiplied by Dy with respect to W2. This is possible because Y is only dependent on the previous layer. Now, let's suppose we don't want to do this for W2 but we want to do it for W1. We can use the exact same process, but now it's one step further. We'll now replace W2 with W1, and we need to apply the same process, or chain rule, yet again to decompose the problem further. Now, we propagate our old gradient that we computed for W2 all the way back one more step to the weight that we're interested in, which is W1. And we keep repeating this process over and over again, propagating these gradients backwards from output to input to compute ultimately what we want in the end: the derivative of every weight so the derivative of our loss with respect to every weight in our neural network. This tells us how much does a small change in every single weight in our network affect the loss? Does our loss go up or down if we change this weight a little bit in this direction or a little bit in that direction?",
+            "There is no functional difference between a neuron and a perceptron; they are the same. Typically, people say \"neural network,\" which is why a single neuron is also gotten popularity, but originally, the term \"perceptron\" is the formal term.",
+            "So, now we've covered a lot. We've covered the forward propagation of information through a neuron and through a neural network all the way through, and we've covered now the back propagation of information to understand how we should change every single one of those weights in our neural network to improve our loss. So, that was the backpropagation algorithm in theory. It's actually pretty simple; it's just a chain rule. There's nothing more than just the chain rule. And the nice part is that deep learning libraries actually do this for you, so they compute backprop for you. You don't actually have to implement it yourself, which is very convenient."
+        ],
+        "paragraph_timestamps": [
+            2872,
+            2918,
+            2951,
+            3033,
+            3127,
+            3142
+        ]
+    },
+    {
+        "num_chapter": 10,
+        "title": "Optimizing Neural Networks",
+        "start_paragraph_number": 112,
+        "end_paragraph_number": 125,
+        "start_time": 3192,
+        "end_time": 3541,
+        "paragraphs": [
+            "But now it's important to touch on the fact that, although the theory behind backpropagation is not that complicated, the practice of optimizing neural networks is a completely different story. It's not straightforward at all, and in practice, it's very difficult and usually very computationally intensive to implement the backpropagation algorithm.",
+            "Here's an illustration from a paper that came out a few years ago, which attempted to visualize a very deep neural network's loss landscape. Previously, we had a depiction of how a neural network would look in a two-dimensional landscape, but real neural networks are not two-dimensional - they're hundreds or millions or billions of dimensions. Now, what would those loss landscapes look like? You can actually try some clever techniques to visualize them. This is one paper that attempted to do that, and it turns out that they look extremely messy.",
+            "The important thing is that, if you do this algorithm and you start in a bad place, depending on your neural network, you may not actually end up in the global solution. So, your initialization matters a lot, and you need to traverse these local minima and try to find the global minimum, or even more than that, you need to construct neural networks that have loss landscapes that are much more amenable to optimization than this one.",
+            "So, this is a very bad loss landscape. There are some techniques that we can apply to our neural networks that smooth out their loss landscapes and make them easier to optimize. Recall that update equation that we talked about earlier with gradient descent - the one with the parameter that we didn't talk about, which we described as the 'little step' that you could take. It's a small number that multiplies with the direction, which is your gradient. It just tells you, 'Okay, I'm not going to just go all the way in this direction; I'll just take a small step in this direction.'",
+            "In practice, even setting this value right is just one number, and setting this one number can be rather difficult. If we set the learning rate too high, we might overshoot the minimum, and if we set it too low, we might not make enough progress. So, finding the right learning rate is crucial for the optimization process.",
+            "Setting the learning rate too small can result in the model getting stuck in local minima, where it converges very slowly. On the other hand, if the learning rate is too large, it can overshoot and even diverge, causing the model to explode and never find a minimum.",
+            "Ideally, we want to use learning rates that are not too small and not too large, so they're large enough to avoid local minima but small enough to find their way into the global minimum. Something like this is what we should intuitively have in mind: a learning rate that can overshoot local minima but find its way into a better minimum and then stabilize itself there.",
+            "So, how do we actually set these learning rates in practice? What does that process look like?",
+            "One idea is to try a bunch of different learning rates and see what works. This is actually a not-bad process in practice, and it's one of the processes that people use. However, let's see if we can do something smarter than this and design algorithms that can adapt to the landscapes.",
+            "In practice, there's no reason why this should be a single number. Can we have learning rates that adapt to the model, the data, the landscapes, and the gradients that it's seeing around? This means that the learning rate may actually increase or decrease as a function of the gradients in the loss function, or how fast we're learning, or many other options.",
+            "There are many different ideas that could be done here, and in fact, there are many widely used different procedures or methodologies for setting the learning rate. During your labs, we actually encourage you to try out some of these different ideas for different types of learning rates and even play around with them. What's the effect of increasing or decreasing your learning rate? You'll see very striking differences. It's because it's on a close interval, why not just find the absolute minimum? You know, test right?",
+            "So, a few things. Number one is that it's not a closed space. Right, so there's an infinite every every weight can be plus or minus up to infinity. Right, so even if it was a one-dimensional neural network with just one weight, it's not a closed space. In practice, it's even worse than that because you have billions of dimensions. Right, so not only is your space your support system in one dimension, it's infinite, but you now have billions of infinite dimensions. Right, or billions of uh infinite support spaces. So, it's not something that you can just like search every weight, every possible weight in your neural in your configuration, or what is every possible weight that this neural network could take and let me test them out. Because it's not practical to do even for a very small neural network in practice.",
+            "So, in your labs, you can really try to put all of this information uh in this picture into practice, which defines your model number one right here. Defines your Optimizer, which previously we denoted as this gradient descent Optimizer. Here, we're calling it uh stochastic gradient descent or SGD. We'll talk about that more in a second. And then also note that your Optimizer, which here we're calling SGD, could be any of these adaptive optimizers. You can swap them out, and you should swap them out. You should test different things here to see the impact of these different methods on your training procedure. And you'll gain very valuable intuition for the different insights that will come with that as well."
+        ],
+        "paragraph_timestamps": [
+            3192,
+            3210,
+            3247,
+            3273,
+            3299,
+            3310,
+            3333,
+            3362,
+            3370,
+            3390,
+            3411,
+            3445,
+            3496
+        ]
+    },
+    {
+        "num_chapter": 11,
+        "title": "Stochastic Gradient Descent and Mini-Batching",
+        "start_paragraph_number": 125,
+        "end_paragraph_number": 131,
+        "start_time": 3541,
+        "end_time": 3748,
+        "paragraphs": [
+            "So, I want to continue briefly for the end of this lecture to discuss tips for training neural networks in practice and how we can focus on the powerful idea of batching data. Right, not seeing all of your data, but rather talking about a topic called batching. So, let's briefly revisit this gradient descent algorithm. The gradient is computed through this gradient computation, and the backprop algorithm I mentioned earlier is a very computationally expensive operation. It's even worse because we previously described it in a way where we would have to compute it over a summation over every single data point in our entire data set. That's how we defined it with the loss function - it's an average over all of our data points, which means that we're summing over all of our data points the gradients.",
+            "In most real-life problems, this would be completely infeasible to do because our data sets are simply too big and the models are too big to compute those gradients on every single iteration. Remember, this isn't just a one-time thing - it's every single step that you do. You keep taking small steps, so you keep needing to repeat this process.",
+            "Instead, let's define a new gradient descent algorithm called SGD, or stochastic gradient descent. Instead of computing the gradient over the entire data set, let's just pick a single training point and compute that one training point's gradient. The nice thing about that is that it's much easier to compute that gradient - it only needs one point. The downside is that it's very noisy, it's very stochastic, since it was computed using just that one example.",
+            "So, what's the middle ground? The middle ground is to take not one data point, nor the full data set, but a batch of data. This is called a mini-batch, which could be something in practice like 32 pieces of data, a common batch size. This gives us an estimate of the true gradient, so we approximate the gradient by averaging the gradient of these 32 samples. It's still fast because 32 is much smaller than the size of your entire data set, but it's pretty quick now. It's still noisy, but it's a good compromise.",
+            "In practice, mini-batches are usually used because you can still iterate much faster, and since B is normally not that large - think of something like in the tens or the hundreds of samples - it's very fast to compute in practice compared to regular gradient descent. It's also much more accurate compared to stochastic gradient descent. The increase in accuracy of this gradient estimation allows us to converge to our solution significantly faster, as well. It's not only about the speed; it's just about the increase in accuracy of those gradients, which allows us to get to our solution much faster, which ultimately means that we can train much faster as well and we can save compute.",
+            "The other really nice thing about mini-batches is that they allow for parallelizing our computation. Right, and that was a concept that we had talked about earlier in the class as well. Here's where it's coming in: we can split up those batches, right? So, those 32 pieces of data - let's say if our batch size is 32 - we can split them up onto different workers, right? Different parts of the GPU can tackle those different parts of our data points. This can allow us to basically achieve even more significant speed-ups using GPU architectures and GPU hardware."
+        ],
+        "paragraph_timestamps": [
+            3541,
+            3589,
+            3609,
+            3643,
+            3673,
+            3714
+        ]
+    },
+    {
+        "num_chapter": 12,
+        "title": "Understanding Overfitting and Regularization Techniques",
+        "start_paragraph_number": 131,
+        "end_paragraph_number": 146,
+        "start_time": 3748,
+        "end_time": 4136,
+        "paragraphs": [
+            "Finally, the last topic I want to talk about before we end this lecture and move on to lecture number two is overfitting. Overfitting is this idea that is actually not a deep learning-centric problem at all; it's a problem that exists in all of machine learning. The key problem is that it's actually one that addresses how you can accurately define if your model is actually capturing your true data set, right? Or if it's just learning kind of the subtle details that are kind of spurious correlating to your data set.",
+            "So, let's say we want to build models that can learn representations from our training data that still generalize to brand new, unseen test points. That's the real goal here - we want to teach our model something based on a lot of training data, but then we don't want it to do well in the training data; we want it to do well when we deploy it into the real world and it's seeing things that it has never seen during training.",
+            "The concept of overfitting is exactly addressing that problem. Overfitting means if your model is doing very well on your training data but very badly in testing, it's that means it's overfitting - it's overfitting to the training data that it saw. On the other hand, there's also underfitting. On the left-hand side, you can see basically not fitting the data enough, which means that you know you're going to achieve very similar performance on your testing distribution, but both are underperforming the actual capabilities of your system.",
+            "Ideally, you want to end up somewhere in the middle - not too complex, where you're memorizing all of the nuances in your training data, like on the right, but you still want to continue to perform well even based on brand new data, so you're not underfitting as well.",
+            "To actually address this problem in neural networks and in machine learning in general, there are a few different ways that you should be aware of and how to do it, because you'll need to apply them as part of your Solutions and your software Labs as well.",
+            "The key concept here is called regularization. Regularization is a technique that you can introduce, and simply put, all regularization is is a way to discourage your model from learning these nuances in your training data - that's all it is. And as we've seen before, it's actually critical for our models to be able to generalize - you know, not just on training data, but really what we care about is the testing data.",
+            "The most popular regularization technique that's important for you to understand is this very simple idea called Dropout. Let's revisit this picture of a deep neural network that we've been discussing.",
+            "In seeing all lectures right, Dropout is a training technique during training. What we're going to do is randomly set some of the activations, or these outputs of every single neuron, to zero. We're just randomly going to set them to zero with some probability. Right, so let's say 50% is our probability. That means that we're going to take all of the activations in our neural network and, with a probability of 50%, before we pass that activation onto the next neuron, we're just going to set it to zero and not pass on anything. So, effectively, 50% of the neurons are going to be kind of shut down or 'killed' in a forward pass, and you're only going to forward pass information with the other 50% of your neurons.",
+            "This idea is extremely powerful, actually, because it lowers the capacity of our neural network. It not only lowers the capacity of our neural network but it's dynamically lowering it because, on the next iteration, we're going to pick a different 50% of neurons that we drop out. So, constantly, the network is going to have to learn to build different pathways from input to output, and it can't rely on any small part of the features that are present in any part of the training dataset too extensively. Right, because it's constantly being forced to find these different pathways with random probabilities.",
+            "So, that's Dropout. The second regularization technique is going to be this notion called early stopping, which is actually something that is model-agnostic. You can apply this to any type of model as long as you have a testing set that you can play around with.",
+            "The idea here is that we have already a pretty formal mathematical definition of what it means to overfit. Overfitting is just when our model starts to perform worse on our test set. That's really all it is. So, what if we plot over the course of training? So, the x-axis is as we're training the model, and let's look at the performance on both the training set and the test set.",
+            "In the beginning, you can see that the model is performing really well on both the training set and the test set. But as we continue training, you can see that the model starts to perform better and better on the training set, but worse and worse on the test set. This is actually a sign of overfitting. So, what we can do is stop training when the model starts to perform worse on the test set. This is called early stopping.",
+            "By stopping training when the model starts to overfit, we can prevent overfitting and ensure that our model generalizes well to new, unseen data. That's because the training set and the test set are both going down, and they continue to go down, which is excellent because it means that our model is getting stronger. Eventually, though, what you'll notice is that the test loss plateaus and starts to increase. On the other hand, the training loss - there's no reason why the training loss should ever need to stop going down, right? Training losses generally always continue to decay as long as there is capacity in the neural network to learn those differences, right?",
+            "But the important point is that this continues for the rest of training, and we want to stop training basically right after this point. This is the really important point because this is where we need to stop training. Right after this point, this is the happy medium because after this point, we start to overfit on parts of the data where our training accuracy becomes actually better than our testing accuracy. So, our testing accuracy is getting worse, but our training accuracy is still improving, which means overfitting.",
+            "On the other hand, on the left-hand side, this is the opposite problem. We have not fully utilized the capacity of our model, and the testing accuracy can still improve further. This is a very powerful idea, but it's actually extremely easy to implement in practice because all you really have to do is just monitor the loss over the course of training, right? And you just have to pick the model where the testing accuracy starts to get worse."
+        ],
+        "paragraph_timestamps": [
+            3748,
+            3791,
+            3814,
+            3849,
+            3864,
+            3877,
+            3905,
+            3918,
+            3964,
+            4000,
+            4011,
+            4021,
+            4031,
+            4068,
+            4098
+        ]
+    },
+    {
+        "num_chapter": 13,
+        "title": "Summary and Transition to Next Lecture",
+        "start_paragraph_number": 146,
+        "end_paragraph_number": 149,
+        "start_time": 4136,
+        "end_time": 4165,
+        "paragraphs": [
+            "I'll conclude this lecture by summarizing three key points we've covered so far. So far, we've learned the fundamental building blocks of neural networks, starting from a single neuron, also called a perceptron. We've seen how we can stack these systems on top of each other to create a hierarchical network, and how we can mathematically optimize those systems.",
+            "We've also covered different types of systems, and then, in the final part of the class, we discussed techniques and tips for training and applying these systems in practice.",
+            "Now, in the next lecture, we'll be hearing from Ava on deep sequence modeling using RNNs and a new and exciting algorithm called the Transformer, which is built on the principle of attention. You'll learn more about it in the next class, but for now, let's take a brief pause and resume in about five minutes so we can switch speakers and Ava can start her presentation. Okay, thank you."
+        ],
+        "paragraph_timestamps": [
+            4136,
+            4156,
+            4165
+        ]
+    }
+]

app.py

@@ -4,133 +4,178 @@ import json
 
 from youtube_transcript_api import YouTubeTranscriptApi
 
-from openai import OpenAI
+import utils
 
-import numpy as np
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.metrics.pairwise import cosine_similarity
+from openai import OpenAI
+from groq import Groq
 
-def gradio_video_id_to_transcript(video_id):
+from dotenv import load_dotenv
+load_dotenv()
+GROQ_API_KEY = os.getenv("GROQ_API_KEY")
+OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
 
-    transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
-    transcript_formatted = [{'start': entry['start'], 'text': entry['text']} for entry in transcript[0:10]]
-    transcript_formatted_str = json.dumps(transcript_formatted, indent=2)+'...'
+#import importlib
+#importlib.reload(utils)
 
-    return {output_transcript: transcript_formatted_str,
-            gv_transcript: transcript}
+def get_llm_client_and_model(llm_model):
+    if llm_model == "llama3-8b":
+        llm_client = Groq(api_key=GROQ_API_KEY)
+        llm_model = 'llama3-8b-8192'
 
-def gradio_transcript_to_paragraphs(gv_transcript_value):
+    elif llm_model == "gpt-4o-mini":
+        llm_client = OpenAI(api_key=OPENAI_API_KEY)
+        llm_model = 'gpt-4o-mini-2024-07-18'
 
-    paragraphs, nb_input_tokens, nb_output_tokens, price = \
-        transcript_to_paragraphs(gv_transcript_value, openai_client, openai_model, chunk_size=5000)
+    return llm_client, llm_model
 
-    paragraphs_formatted_str = json.dumps(paragraphs[0:4], indent=2)+'...'
 
-    return {output_paragraphs: paragraphs_formatted_str,
-            gv_paragraphs: paragraphs}
+def gradio_process_video(video_id,
+                         model_format_transcript, model_toc,
+                         chunk_size_format_transcript, chunk_size_toc,
+                         progress=gr.Progress()):
+    if video_id in ["ErnWZxJovaM"]:
+        chapters = utils.load_json_chapters(video_id)
 
-def gradio_paragraphs_to_toc(gv_paragraphs_value):
+    else:
 
-    paragraphs_dict = gv_paragraphs_value
+        transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
 
-    json_toc, nb_input_tokens, nb_output_tokens, price = \
-      paragraphs_to_toc(paragraphs_dict, openai_client, openai_model, chunk_size=100)
+        chunk_size_format_transcript = int(chunk_size_format_transcript)
 
-    json_toc_formatted_str = json.dumps(json_toc[0:4], indent=2)+'...'
+        llm_client_format_transcript, llm_model_format_transcript = \
+            get_llm_client_and_model(model_format_transcript)
 
-    return {output_toc: json_toc_formatted_str,
-            gv_toc: json_toc}
+        paragraphs, nb_input_tokens, nb_output_tokens, price = \
+            utils.transcript_to_paragraphs(transcript, \
+                                           llm_client_format_transcript, llm_model_format_transcript, \
+                                           chunk_size=chunk_size_format_transcript, progress=progress)
 
+        paragraphs = utils.add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50)
 
-def gradio_get_paragraphs_timestamps(gv_transcript_value, gv_paragraphs_value):
+        chunk_size_toc = int(chunk_size_toc)
 
-    paragraphs = add_timestamps_to_paragraphs(gv_transcript_value, gv_paragraphs_value, num_words=50)
+        llm_client_get_toc, llm_model_get_toc = \
+            get_llm_client_and_model(model_toc)
 
-    paragraphs_formatted_str = json.dumps(paragraphs[0:4], indent=2)+'...'
+        json_toc, nb_input_tokens, nb_output_tokens, price = \
+            utils.paragraphs_to_toc(paragraphs, \
+                                    llm_client_get_toc, llm_model_get_toc, \
+                                    chunk_size=chunk_size_toc)
 
-    return {output_paragraphs_timestamps: paragraphs_formatted_str,
-            gv_paragraphs: paragraphs}
+        chapters = utils.get_chapters(paragraphs, json_toc)
 
+    output_html = utils.get_result_as_html(chapters, video_id)
 
-def gradio_get_chapters(gv_paragraphs_value, gv_toc_value):
+    return {output_processing: str(output_html),
+            gv_output: output_html}
 
-    chapters = get_chapters(gv_paragraphs_value, gv_toc_value)
 
-    chapters_formatted_str = json.dumps(chapters[0:4], indent=2)+'...'
+def gradio_process_video(video_id,
+                         model_format_transcript, model_toc,
+                         chunk_size_format_transcript, chunk_size_toc,
+                         progress=gr.Progress()):
+    if video_id in ["ErnWZxJovaM"]:
+        chapters = utils.load_json_chapters(video_id)
 
-    return {output_chapters: chapters_formatted_str,
-            gv_chapters: chapters}
+    else:
 
+        transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
 
-def gradio_get_markdown(gv_chapters_value):
+        chunk_size_format_transcript = int(chunk_size_format_transcript)
 
-    markdown = chapters_to_markdown(gv_chapters_value)
+        llm_client_format_transcript, llm_model_format_transcript = \
+            get_llm_client_and_model(model_format_transcript)
 
-    return markdown
+        paragraphs, nb_input_tokens, nb_output_tokens, price = \
+            utils.transcript_to_paragraphs(transcript, \
+                                           llm_client_format_transcript, llm_model_format_transcript, \
+                                           chunk_size=chunk_size_format_transcript, progress=progress)
 
-with gr.Blocks() as app:
+        paragraphs = utils.add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50)
 
-    gr.Markdown("## Get transcript")
+        chunk_size_toc = int(chunk_size_toc)
 
-    gv_transcript = gr.State()
-    video_id_input = gr.Textbox(label="Video ID", value = "ErnWZxJovaM")
-    get_transcript_button = gr.Button("Get transcript")
-    output_transcript = gr.Textbox(label = "Transcript (JSON format - start, text)")
+        llm_client_get_toc, llm_model_get_toc = \
+            get_llm_client_and_model(model_toc)
 
-    get_transcript_button.click(gradio_video_id_to_transcript,
-                                inputs=[video_id_input],
-                                outputs=[output_transcript, gv_transcript])
+        json_toc, nb_input_tokens, nb_output_tokens, price = \
+            utils.paragraphs_to_toc(paragraphs, \
+                                    llm_client_get_toc, llm_model_get_toc, \
+                                    chunk_size=chunk_size_toc)
 
-    gr.Markdown("## Transcript to paragraphs")
+        chapters = utils.get_chapters(paragraphs, json_toc)
 
-    gv_paragraphs = gr.State()
-    get_paragraphs_button = gr.Button("Get paragraphs")
-    output_paragraphs = gr.Textbox(label = "Paragraphs (JSON format - paragraph_number, paragraph_text)")
+    output_html = utils.get_result_as_html(chapters, video_id)
 
-    get_paragraphs_button.click(gradio_transcript_to_paragraphs,
-                                inputs=[gv_transcript],
-                                outputs=[output_paragraphs, gv_paragraphs])
+    return {output_processing: str(output_html),
+            gv_output: output_html}
 
-    gr.Markdown("## Get table of content")
 
-    gv_toc = gr.State()
-    get_toc_button = gr.Button("Get table of contents")
-    output_toc = gr.Textbox(label = "Table of content (JSON format - paragraph_number, title)")
+# %%
+css = """
+.content {
+    padding: 20px;
+    max-width: 800px;
+    margin: 0 auto;
+    background-color: #ffffff;
+    box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
+    border-radius: 8px;
+}
+"""
 
-    get_toc_button.click(gradio_paragraphs_to_toc,
-                         inputs=[gv_paragraphs],
-                         outputs=[output_toc, gv_toc])
+example_video_id = "ErnWZxJovaM"
+example_chapters = utils.load_json_chapters(example_video_id)
+example_output_html = utils.get_result_as_html(example_chapters, example_video_id)
 
+with (gr.Blocks(css=css) as app):
+    gr.HTML("<div align='center'><h1 class='header'>Demo: Automatic video chaptering with LLMs and TF-IDF</h1></div>")
+    gr.HTML("<div align='center'><h3 class='header'>From raw transcript to structured document</h3></div>")
+    gr.HTML("<hr>")
+    gr.Markdown("""This demo relies on 
+                - Groq's Llama 3 8B for transcript preprocessing
+                - OpenAI's GPT-4o-mini for chaptering. Note: Using GPT-4o-mini for transcript preprocessing will improve results, but takes longer (around 2/3 minutes for a one-hour video)
 
-    gr.Markdown("## Infer paragraph timestamps with TF-IDF")
+                The following YouTube video ID are already preprocessed (copy and paste ID in box below): 
 
-    get_timestamps_button = gr.Button("Infer paragraph timestamps")
-    output_paragraphs_timestamps = gr.Textbox(label = "Paragraphs (JSON format - paragraph_number, paragraph_text, start)")
+                - `ErnWZxJovaM`: [MIT course](https://www.youtube.com/watch?v=ErnWZxJovaM)
+                - `EuC1GWhQdKE`: [Anthropic](https://www.youtube.com/watch?v=EuC1GWhQdKE)
 
-    get_timestamps_button.click(gradio_get_paragraphs_timestamps,
-                                inputs=[gv_transcript, gv_paragraphs],
-                                outputs=[output_paragraphs_timestamps, gv_paragraphs])
+                Check the [Medium article]() for more details"""
+                )
 
-    gr.Markdown("## Get chapters")
+    gv_transcript = gr.State()
 
+    video_id_input = gr.Textbox(label="Enter YouTube Video ID", value="EuC1GWhQdKE")
 
-    gv_chapters = gr.State()
-    get_chapters_button = gr.Button("Get chapters")
-    output_chapters = gr.Textbox(label = "Chapters (JSON format)")
+    with gr.Accordion("Set parameters", open=False):
+        with gr.Row():
+            with gr.Column(scale=1):
+                model_format_transcript = gr.Dropdown(
+                    [("LLama 3 8B (Groq)", "llama3-8b"), ("GPT-4o-mini (OpenAI)", "gpt-4o-mini")],
+                    label="Transcript preprocessing", value="llama3-8b", interactive=True)
+                chunk_size_format_transcript = gr.Textbox(label="Preprocessing chunk size", value=2000)
+            with gr.Column(scale=1):
+                model_toc = gr.Dropdown([("LLama 3 8B (Groq)", "llama3-8b"), ("GPT-4o-mini (OpenAI)", "gpt-4o-mini")],
+                                        label="Chaptering", value="gpt-4o-mini", interactive=True)
+                chunk_size_toc = gr.Textbox(label="Chaptering chunk size", value=30)
+            with gr.Column(scale=1):
+                api_key_openai = gr.Textbox(label="OpenAI API Key", value="xxx")
+                api_key_groq = gr.Textbox(label="Groq API Key", value="xxx")
 
-    get_chapters_button.click(gradio_get_chapters,
-                              inputs=[gv_paragraphs, gv_toc],
-                              outputs=[output_chapters, gv_chapters])
+    processing_button = gr.Button("Process transcript")
 
+    gv_output = gr.State()
 
-    gr.Markdown("## Markdown formatting")
+    gr.HTML("<hr>")
 
-    get_markdown_button = gr.Button("Markdown formatting")
-    output_markdown = gr.Markdown(label = "Chapters (Markdown format)")
+    output_processing = gr.HTML(label="Output processing", value=example_output_html)
 
-    get_markdown_button.click(gradio_get_markdown,
-                              inputs=[gv_chapters],
-                              outputs=[output_markdown])
+    processing_button.click(gradio_process_video,
+                            inputs=[video_id_input,
+                                    model_format_transcript, model_toc,
+                                    chunk_size_format_transcript, chunk_size_toc],
+                            outputs=[output_processing, gv_output])
 
+    # gr.HTML(result_as_html)
 
-app.launch(debug=True)
+app.launch(debug=True, width="100%")

utils.py

@@ -0,0 +1,426 @@
+import json
+import re
+
+import numpy as np
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+
+
+########################### LLM call ###########################
+
+price_token={'gpt-4o': {'input': 5/1000000, 'output': 15/1000000},
+             'gpt-4o-2024-08-06': {'input': 2.5/1000000, 'output': 10/1000000},
+             'gpt-4o-mini-2024-07-18': {'input': 0.15/1000000, 'output': 0.6/1000000},
+             'llama3-8b-8192' : {'input': 0.05 / 1000000, 'output': 0.08 / 1000000},
+             'llama3-70b-8192' : {'input': 0.59 / 1000000, 'output': 0.79 / 1000000},
+             'claude-3-5-sonnet-20240620': {'input': 3/1000000, 'output': 15/1000000},
+             'claude-3-haiku-20240307': {'input': 0.25/1000000, 'output': 1.25/1000000},
+             }
+def call_llm(client, model, system_prompt, prompt,
+             temperature=0, seed=42, response_format=None, max_tokens=5000):
+
+    response = client.chat.completions.create(
+        messages=[
+            {
+                "role": "system",
+                "content": system_prompt
+            },
+            {
+                "role": "user",
+                "content": prompt
+            }
+        ],
+        model=model,
+        temperature=temperature,
+        seed=seed,
+        response_format=response_format,
+        max_tokens=max_tokens
+    )
+
+    nb_input_tokens = response.usage.prompt_tokens
+    nb_output_tokens = response.usage.completion_tokens
+    price = nb_input_tokens * price_token[model]['input'] + nb_output_tokens * price_token[model]['output']
+
+    print(f"input tokens: {nb_input_tokens}; output tokens: {nb_output_tokens}, price: {price}")
+
+    response_content=response.choices[0].message.content
+
+    return response_content, nb_input_tokens, nb_output_tokens, price
+
+########################### Step 2: Transcript to paragraph ###########################
+
+system_prompt_transcript_to_paragraphs = f"""
+
+You are a helpful assistant.
+
+Your task is to improve the user input's readability: add punctuation if needed, remove verbal tics, correct grammatical errors, and add appropriate line breaks with '\n\n'.
+
+Put your answer within <answer></answer> tags.
+
+"""
+
+
+
+def transcript_to_paragraphs(transcript, llm_client, llm_model, chunk_size=5000, progress=None):
+
+    transcript_as_text = ' '.join([s['text'] for s in transcript])
+
+    paragraphs = []
+    last_paragraph = ""
+
+    total_nb_input_tokens, total_nb_output_tokens, total_price = 0, 0, 0
+
+    nb_chunks = int(len(transcript_as_text) / chunk_size) + 1
+    progress_i = 0
+    print(f"Number of chunks: {nb_chunks}")
+
+    # for i in range(0, 10000, chunk_size):
+    for i in range(0, len(transcript_as_text), chunk_size):
+
+        print("i is: " + str(i))
+
+        chunk = last_paragraph + " " + transcript_as_text[i:i + chunk_size]
+
+        if progress is not None:
+            progress_i += 1
+            progress(progress_i / nb_chunks, desc="Processing")
+
+        found_edited_transcript = False
+
+        while not found_edited_transcript:
+
+            response_content, nb_input_tokens, nb_output_tokens, price = \
+                call_llm(llm_client, llm_model,
+                         system_prompt=system_prompt_transcript_to_paragraphs, prompt=chunk,
+                         temperature=0.2, seed=42, response_format=None)
+
+            if not "</answer>" in response_content:
+                response_content += "</answer>"
+
+            # Extract content from <edited_transcript> tags
+            pattern = re.compile(r'<answer>(.*?)</answer>', re.DOTALL)
+            response_content_edited = pattern.findall(response_content)
+
+            if len(response_content_edited) > 0:
+                found_edited_transcript = True
+                response_content_edited = response_content_edited[0]
+
+            else:
+                print("No edited transcript found. Trying again.")
+                print(response_content[0:100])
+                print(response_content[-100:])
+
+        total_nb_input_tokens += nb_input_tokens
+        total_nb_output_tokens += nb_output_tokens
+        total_price += price
+
+        paragraphs_chunk = response_content_edited.strip().split('\n\n')
+
+        print('Found paragraphs:', len(paragraphs_chunk))
+        last_paragraph = paragraphs_chunk[-1]
+
+        paragraphs += paragraphs_chunk[:-1]
+
+    paragraphs += [last_paragraph]
+
+    paragraphs_dict = [{'paragraph_number': i, 'paragraph_text': paragraph} for i, paragraph in enumerate(paragraphs)]
+
+    return paragraphs_dict, total_nb_input_tokens, total_nb_output_tokens, total_price
+
+########################### Step 3: Infer timestamps ###########################
+
+def transform_text_segments(text_segments, num_words=50):
+    # Initialize variables
+    transformed_segments = []
+    current_index = 0
+    num_segments = len(text_segments)
+
+    for i in range(num_segments):
+
+        current_index = i
+
+        # Get the current segment's starting timestamp and text
+        current_segment = text_segments[current_index]
+        current_text = current_segment['text']
+
+        # Initialize a list to hold the combined text
+        combined_text = " ".join(current_text.split()[:num_words])
+        number_words_collected = len(current_text.split())
+
+        # Collect words from subsequent segments
+        while number_words_collected < num_words and (current_index + 1) < num_segments:
+            current_index += 1
+            next_segment = text_segments[current_index]
+            next_text = next_segment['text']
+            next_words = next_text.split()
+
+            # Append words from the next segment
+            if number_words_collected + len(next_words) <= num_words:
+                combined_text += ' ' + next_text
+                number_words_collected += len(next_words)
+            else:
+                # Only append enough words to reach the num_words limit
+                words_needed = num_words - number_words_collected
+                combined_text += ' ' + ' '.join(next_words[:words_needed])
+                number_words_collected = num_words
+
+        # Append the combined segment to the result
+        transformed_segments.append(combined_text)
+
+    return transformed_segments
+
+
+def add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50):
+    list_indices = []
+
+    transcript_num_words = transform_text_segments(transcript, num_words=num_words)
+
+    paragraphs_start_text = [{"start": p['paragraph_number'], "text": p['paragraph_text']} for p in paragraphs]
+    paragraphs_num_words = transform_text_segments(paragraphs_start_text, num_words=num_words)
+
+    # Create a TF-IDF vectorizer
+    vectorizer = TfidfVectorizer().fit_transform(transcript_num_words + paragraphs_num_words)
+    # Get the TF-IDF vectors for the transcript and the excerpt
+    vectors = vectorizer.toarray()
+
+    for i in range(len(paragraphs_num_words)):
+
+        # Extract the TF-IDF vector for the paragraph
+        paragraph_vector = vectors[len(transcript_num_words) + i]
+
+        # Calculate the cosine similarity between the paragraph vector and each transcript chunk
+        similarities = cosine_similarity(vectors[:len(transcript_num_words)], paragraph_vector.reshape(1, -1))
+        # Find the index of the most similar chunk
+        best_match_index = int(np.argmax(similarities))
+
+        list_indices.append(best_match_index)
+
+        paragraphs[i]['matched_index'] = best_match_index
+        paragraphs[i]['matched_text'] = transcript[best_match_index]['text']
+        paragraphs[i]['start_time'] = int(transcript[best_match_index]['start']) - 2
+        if paragraphs[i]['start_time'] < 0:
+            paragraphs[i]['start_time'] = 0
+
+    return paragraphs
+
+########################### Step 4: Generate table of content ###########################
+
+
+system_prompt_paragraphs_to_toc = """
+
+	You are a helpful assistant.
+
+	You are given a transcript of a course in JSON format as a list of paragraphs, each containing 'paragraph_number' and 'paragraph_text' keys.
+
+	Your task is to group consecutive paragraphs in chapters for the course and identify meaningful chapter titles.
+
+	Here are the steps to follow:
+
+1. Read the transcript carefully to understand its general structure and the main topics covered.
+2. Look for clues that a new chapter is about to start. This could be a change of topic, a change of time or setting, the introduction of new themes or topics, or the speaker's explicit mention of a new part.
+3. For each chapter, keep track of the paragraph number that starts the chapter and identify a meaningful chapter title.
+4. Chapters should ideally be equally spaced throughout the transcript, and discuss a specific topic.
+5. A chapter MUST have more than 4 paragraphs.
+
+	Format your result in JSON, with a list dictionaries for chapters, with 'start_paragraph_number':integer and 'title':string as key:value.
+
+	Example: 
+    {"chapters": 
+        [{"start_paragraph_number": 0, "title": "Introduction"}, 
+         {"start_paragraph_number": 10, "title": "Chapter 1"}
+        ]
+    }
+
+"""
+
+
+def paragraphs_to_toc(paragraphs, llm_client, llm_model, chunk_size=100):
+    chapters = []
+    number_last_chapter = 0
+
+    total_nb_input_tokens, total_nb_output_tokens, total_price = 0, 0, 0
+
+    while number_last_chapter < len(paragraphs):
+
+        print(number_last_chapter)
+
+        chunk = paragraphs[number_last_chapter:(number_last_chapter + chunk_size)]
+        chunk = [{'paragraph_number': p['paragraph_number'], 'paragraph_text': p['paragraph_text']} for p in chunk]
+
+        chunk_json_dump = json.dumps(chunk)
+
+        content, nb_input_tokens, nb_output_tokens, price = call_llm( \
+            llm_client, llm_model, \
+            system_prompt_paragraphs_to_toc, chunk_json_dump, \
+            temperature=0, seed=42, response_format={"type": "json_object"})
+
+        total_nb_input_tokens += nb_input_tokens
+        total_nb_output_tokens += nb_output_tokens
+
+        chapters_chunk = json.loads(content)['chapters']
+
+        if number_last_chapter == chapters_chunk[-1]['start_paragraph_number']:
+            break
+
+        chapters += chapters_chunk[:-1]
+
+        number_last_chapter = chapters_chunk[-1]['start_paragraph_number']
+        if number_last_chapter >= len(paragraphs) - 5:
+            break
+
+    total_price = (total_nb_input_tokens * price_token[llm_model]['input'] +
+                   total_nb_output_tokens * price_token[llm_model]['output'])
+
+    chapters += [chapters_chunk[-1]]
+
+    return chapters, total_nb_input_tokens, total_nb_output_tokens, total_price
+
+
+########################### Step 5: Chapter rendering functions ###########################
+
+def get_chapters(paragraphs, table_of_content):
+
+    chapters = []
+
+    for i in range(len(table_of_content)):
+
+
+        if i < len(table_of_content) - 1:
+
+            chapter = {'num_chapter': i,
+                       'title': table_of_content[i]['title'],
+                       'start_paragraph_number': table_of_content[i]['start_paragraph_number'],
+                       'end_paragraph_number': table_of_content[i + 1]['start_paragraph_number'],
+                       'start_time': paragraphs[table_of_content[i]['start_paragraph_number']]['start_time'],
+                       'end_time': paragraphs[table_of_content[i + 1]['start_paragraph_number']]['start_time'],
+                      }
+
+        else:
+            chapter = {'num_chapter': i,
+                       'title': table_of_content[i]['title'],
+                       'start_paragraph_number': table_of_content[i]['start_paragraph_number'],
+                       'end_paragraph_number': len(paragraphs),
+                       'start_time': paragraphs[table_of_content[i]['start_paragraph_number']]['start_time'],
+                       'end_time': paragraphs[-1]['start_time'],
+                      }
+
+        paragraphs_chapter = [paragraphs[j]['paragraph_text'] for j in
+                                range(chapter['start_paragraph_number'], chapter['end_paragraph_number'])]
+
+        paragraph_timestamps_chapter = [paragraphs[j]['start_time'] for j in
+                                          range(chapter['start_paragraph_number'], chapter['end_paragraph_number'])]
+
+        chapter['paragraphs'] = paragraphs_chapter
+        chapter['paragraph_timestamps'] = paragraph_timestamps_chapter
+
+        chapters.append(chapter)
+
+    return chapters
+
+def convert_seconds_to_hms(seconds):
+    # Calculate hours, minutes, and remaining seconds
+    hours = seconds // 3600
+    minutes = (seconds % 3600) // 60
+    remaining_seconds = seconds % 60
+
+    # Format the result as HH:MM:SS
+    return f"{hours:02}:{minutes:02}:{remaining_seconds:02}"
+
+def toc_to_html(chapters):
+
+    toc_html = "<h1>Video chapters</h1><p>\n"
+
+    for chapter in chapters:
+        num_chapter = chapter['num_chapter']
+        title = chapter['title']
+
+        from_to = convert_seconds_to_hms(int(chapter['start_time'])) + " - "
+
+        toc_html += f"""{from_to}<a href = "#{num_chapter}" >{num_chapter+1} - {title}</a><br>\n"""
+
+    return toc_html
+
+
+def section_to_html(section_json_data):
+    formatted_section = ""
+
+    paragraphs = section_json_data['paragraphs']
+    paragraphs_timestamp_hms = [convert_seconds_to_hms(int(section_json_data['paragraph_timestamps'][i])) for i in range(len(paragraphs))]
+
+    for i, (paragraph, paragraph_timestamp_hms) in enumerate(zip(paragraphs, paragraphs_timestamp_hms)):
+
+        formatted_section += f"""
+        <div class="row mb-4">
+            <div class="col-md-1">
+                {paragraph_timestamp_hms}
+            </div>
+            <div class="col-md-11">
+                <p>{paragraph}</p>
+            </div>
+        </div>"""
+
+    num_section = section_json_data['num_chapter']
+
+    from_to = "From "+convert_seconds_to_hms(int(section_json_data['start_time'])) + " to " + convert_seconds_to_hms(
+        int(section_json_data['end_time']))
+
+    title = f"{section_json_data['title']}"
+
+    title_link = f"""<div class="transcript-title-icon" " id="{num_section}">{num_section+1} - {title}</div>"""
+
+    summary_section = f"""
+            <h2>{title_link}</h2>
+            {from_to}
+            <p>
+            <div class="summary-section">
+                <div class="summary-text" >
+                    {formatted_section}
+                </div>
+            </div>
+            """
+
+    return summary_section
+
+
+def get_result_as_html(chapters, video_id):
+    video_embed = f"""
+<iframe width="100%" height="400" src="https://www.youtube.com/embed/{video_id}" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+"""
+
+    toc = toc_to_html(chapters)
+
+    edited_transcript = f"""
+<h1>Structured transcript</h1>
+<p>
+"""
+
+    for i in range(len(chapters)):
+        chapter_json_data = chapters[i]
+
+        edited_transcript += section_to_html(chapter_json_data)
+
+    result_as_html = f"""
+<link href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css" rel="stylesheet">
+<div class="container mt-4">
+    <div class="content">
+        {video_embed}
+    </div>
+    <p>
+    <div class="content">
+        {toc}
+    </div>
+    <p>
+    <div class="content">
+        {edited_transcript}
+    </div>
+</div>"""
+
+    return result_as_html
+
+def load_json_chapters(video_id):
+    file_name = f"{video_id}.json"
+    with open(file_name, 'r') as file:
+        chapters = json.load(file)
+
+    return chapters
+