Learn how to build a web scraper ⛏️ with NodeJS using two distinct strategies, including (1) a metatag link preview generator and (2) a fully-interactive bot for Instagram. https://fireship.io/lessons/web-scraping-guide/

1.  Build a simple link preview scraper with Cheerio.

https://github.com/cheeriojs/cheerio

2. Build an interactive Instagram scraper with puppeteer. 

https://github.com/GoogleChrome/puppeteer

#javascript #nodejs #webdev

Take the quizzes 🤓

iOS https://itunes.apple.com/us/app/fireship/id1462592372?mt=8
Android https://play.google.com/store/apps/details?id=io.fireship.quizapp

Upgrade to Fireship PRO at https://fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.
2024-02-04T06:37:51.831Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
3605c72c-9174-4864-a48e-afbb86d144b7
63dd191c2c2156469ca0f8a94bf3e477298ba8d4b14be43e4fc529f66dfef058
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
Today's video is all about extracting data from the Web.
In a perfect world, every website provides an easy to use API that provides access to the data that you want.
But unfortunately, we don't live in a perfect world.
But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force.
And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video.
For example, we'll build a Node.
Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user.
If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video.
But there's another one going out with this video, all you have to do is leave a comment below.
Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node.
Js.
The first use case that we'll look at is building Link Previews in a front end application.
You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL.
We can implement this feature from scratch using a Node.
Js cloud function.
And the reason we need to do this with Node.
Js and not do it entirely from the front end is because the browser will block cross origin requests.
So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser.
One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability.
It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device.
But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman.
On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment.
And this is where the web scraping comes in.
Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image.
So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project.
Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO.
After initializing Firebase functions, you'll notice that we have a few dependencies in our package.
Json.
First, we have Cheerio, which is a Node.
Js implementation of jQuery, which just makes it a lot easier to scrape web pages.
Then we have CORS, so we can specify which URLs have access to this function.
Get URLs is a utility for extracting URLs from a string.
And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser.
And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node.
Js, but we won't get into that one until we get to the second function to scrape Instagram.
Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint.
And CORS is only required if you're building an HTTP function.
If you're using a Firestore function or a callable function, you can just keep it altogether.
And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it.
The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility.
Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all.
Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests.
And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed.
So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript.
And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered.
Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML.
From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom.
If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector.
So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text.
And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview.
But most of the data that we want will come from meta tags.
For example, to show the site description, will grab the description meta tag and then grab the content attribute.
But the thing about meta tags is that there might be multiple formats included on a single website.
Developers might include tags for Open Graph or Twitter.
And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview.
So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag.
You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise.
So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview.
Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function.
I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger.
We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application.
The front end application will send a JSON string as the request body.
So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function.
The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website.
And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally.
I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter.
As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites.
And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview.
Now let's take a second to recap the pros and cons of the code we just implemented.
Cheerio along with Node.
Fetch make it easy to parse data from a pre rendered or server rendered HTML page.
But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load.
If you go to any website, you can see the initially rendered content by viewing the page source.
If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen.
That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact.
And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript.
Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript.
And a great tool for doing that is Puppeteer.
Puppeteer makes it easy to programmatically work with the Chrome browser on the server.
Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that.
Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument.
The username is the actual Instagram page that we want to scrape images from.
Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram.
Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall.
So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page.
Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot.
Then you can use the page type method to select an element from the DOM and simulate typing characters into that form.
If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument.
From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in.
So basically, you have unlimited possibilities for interacting with the website.
Now after you've interacted with something, you might want to wait for the page to update.
You can do this manually by saying wait for and then passing in a number of milliseconds to wait.
Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM.
Now if we want to actually scrape some data at any given time, We can use page evaluate.
And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer.
In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML.
The code you write inside the evaluate callback is like being inside the actual browser.
So you can use native browser APIs like document, query selector, all to grab all the images.
And we can just map them down to the image source as an array of strings and return that from the evaluate call back.
At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client.
And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app.
Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process.
So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web.
I'm I'm gonna go ahead and wrap things up there.
If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip.
Io to get access to even more content.
Thanks for watching, and I will talk to you soon.

Speaker 0: Today's video is all about extracting data from the Web. In a perfect world, every website provides an easy to use API that provides access to the data that you want. But unfortunately, we don't live in a perfect world. But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force. And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video.

For example, we'll build a Node. Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user. If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video. But there's another one going out with this video, all you have to do is leave a comment below. Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node.

Js. The first use case that we'll look at is building Link Previews in a front end application. You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL. We can implement this feature from scratch using a Node. Js cloud function.

And the reason we need to do this with Node. Js and not do it entirely from the front end is because the browser will block cross origin requests. So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser. One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability. It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device.

But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman. On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment. And this is where the web scraping comes in. Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image. So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project.

Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO. After initializing Firebase functions, you'll notice that we have a few dependencies in our package. Json. First, we have Cheerio, which is a Node. Js implementation of jQuery, which just makes it a lot easier to scrape web pages.

Then we have CORS, so we can specify which URLs have access to this function. Get URLs is a utility for extracting URLs from a string. And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser. And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node. Js, but we won't get into that one until we get to the second function to scrape Instagram.

Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint. And CORS is only required if you're building an HTTP function. If you're using a Firestore function or a callable function, you can just keep it altogether. And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it. The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility.

Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all. Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests. And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed. So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript. And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered.

Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML. From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom. If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector. So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text. And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview.

But most of the data that we want will come from meta tags. For example, to show the site description, will grab the description meta tag and then grab the content attribute. But the thing about meta tags is that there might be multiple formats included on a single website. Developers might include tags for Open Graph or Twitter. And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview.

So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag. You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise. So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview. Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function. I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger.

We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application. The front end application will send a JSON string as the request body. So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function. The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website. And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally.

I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter. As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites. And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview. Now let's take a second to recap the pros and cons of the code we just implemented. Cheerio along with Node.

Fetch make it easy to parse data from a pre rendered or server rendered HTML page. But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load. If you go to any website, you can see the initially rendered content by viewing the page source. If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen. That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact.

And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript. Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript. And a great tool for doing that is Puppeteer. Puppeteer makes it easy to programmatically work with the Chrome browser on the server. Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that.

Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument. The username is the actual Instagram page that we want to scrape images from. Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram. Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall. So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page.

Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot. Then you can use the page type method to select an element from the DOM and simulate typing characters into that form. If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument. From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in. So basically, you have unlimited possibilities for interacting with the website.

Now after you've interacted with something, you might want to wait for the page to update. You can do this manually by saying wait for and then passing in a number of milliseconds to wait. Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM. Now if we want to actually scrape some data at any given time, We can use page evaluate. And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer.

In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML. The code you write inside the evaluate callback is like being inside the actual browser. So you can use native browser APIs like document, query selector, all to grab all the images. And we can just map them down to the image source as an array of strings and return that from the evaluate call back. At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client.

And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app. Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process. So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web. I'm I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip.

Io to get access to even more content. Thanks for watching, and I will talk to you soon.
Today's video is all about extracting data from the Web. In a perfect world, every website provides an easy to use API that provides access to the data that you want. But unfortunately, we don't live in a perfect world. But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force.
youtube
internet
google
streaming
And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video. For example, we'll build a Node. Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user.
instagram
youtube
If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video. But there's another one going out with this video, all you have to do is leave a comment below. Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node. Js. The first use case that we'll look at is building Link Previews in a front end application.
You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL. We can implement this feature from scratch using a Node. Js cloud function. And the reason we need to do this with Node.
twitter
Js and not do it entirely from the front end is because the browser will block cross origin requests. So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser. One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability. It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device. But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman. On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment. And this is where the web scraping comes in. Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image.
So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project. Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO.
video gaming
animation
software
software development
youtube
After initializing Firebase functions, you'll notice that we have a few dependencies in our package. Json. First, we have Cheerio, which is a Node. Js implementation of jQuery, which just makes it a lot easier to scrape web pages. Then we have CORS, so we can specify which URLs have access to this function. Get URLs is a utility for extracting URLs from a string. And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser. And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node. Js, but we won't get into that one until we get to the second function to scrape Instagram. Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint. And CORS is only required if you're building an HTTP function. If you're using a Firestore function or a callable function, you can just keep it altogether. And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it. The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility. Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all. Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests.
And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed. So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript.
animation
And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered. Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML. From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom. If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector. So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text. And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview. But most of the data that we want will come from meta tags. For example, to show the site description, will grab the description meta tag and then grab the content attribute. But the thing about meta tags is that there might be multiple formats included on a single website. Developers might include tags for Open Graph or Twitter. And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview. So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag. You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise. So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview. Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function. I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger. We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application. The front end application will send a JSON string as the request body. So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function. The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website. And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally. I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter.
As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites. And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview.
twitter
Now let's take a second to recap the pros and cons of the code we just implemented. Cheerio along with Node. Fetch make it easy to parse data from a pre rendered or server rendered HTML page. But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load. If you go to any website, you can see the initially rendered content by viewing the page source. If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen. That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact. And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript. Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript. And a great tool for doing that is Puppeteer. Puppeteer makes it easy to programmatically work with the Chrome browser on the server. Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that.
Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument. The username is the actual Instagram page that we want to scrape images from. Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram. Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall.
instagram
So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page. Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot. Then you can use the page type method to select an element from the DOM and simulate typing characters into that form. If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument.
instagram
From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in. So basically, you have unlimited possibilities for interacting with the website. Now after you've interacted with something, you might want to wait for the page to update. You can do this manually by saying wait for and then passing in a number of milliseconds to wait. Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM. Now if we want to actually scrape some data at any given time, We can use page evaluate. And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer.
In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML. The code you write inside the evaluate callback is like being inside the actual browser. So you can use native browser APIs like document, query selector, all to grab all the images.
programming
And we can just map them down to the image source as an array of strings and return that from the evaluate call back. At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client.
And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app. Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process. So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web. I'm I'm gonna go ahead and wrap things up there.
instagram
If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip.
Io to get access to even more content. Thanks for watching, and I will talk to you soon.
Today's video is all about extracting data from the Web. In a perfect world, every website provides an easy to use API that provides access to the data that you want. But unfortunately, we don't live in a perfect world. But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force. And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video. For example, we'll build a Node. Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user. If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video. But there's another one going out with this video, all you have to do is leave a comment below. Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node. Js. The first use case that we'll look at is building Link Previews in a front end application. You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL. We can implement this feature from scratch using a Node. Js cloud function. And the reason we need to do this with Node. Js and not do it entirely from the front end is because the browser will block cross origin requests. So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser. One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability. It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device. But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman. On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment. And this is where the web scraping comes in. Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image. So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project. Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO. After initializing Firebase functions, you'll notice that we have a few dependencies in our package. Json. First, we have Cheerio, which is a Node. Js implementation of jQuery, which just makes it a lot easier to scrape web pages. Then we have CORS, so we can specify which URLs have access to this function. Get URLs is a utility for extracting URLs from a string. And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser. And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node. Js, but we won't get into that one until we get to the second function to scrape Instagram. Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint. And CORS is only required if you're building an HTTP function. If you're using a Firestore function or a callable function, you can just keep it altogether. And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it. The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility. Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all. Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests. And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed. So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript. And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered. Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML. From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom. If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector. So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text. And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview. But most of the data that we want will come from meta tags. For example, to show the site description, will grab the description meta tag and then grab the content attribute. But the thing about meta tags is that there might be multiple formats included on a single website. Developers might include tags for Open Graph or Twitter. And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview. So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag. You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise. So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview. Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function. I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger. We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application. The front end application will send a JSON string as the request body. So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function. The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website. And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally. I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter. As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites. And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview. Now let's take a second to recap the pros and cons of the code we just implemented. Cheerio along with Node. Fetch make it easy to parse data from a pre rendered or server rendered HTML page. But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load. If you go to any website, you can see the initially rendered content by viewing the page source. If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen. That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact. And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript. Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript. And a great tool for doing that is Puppeteer. Puppeteer makes it easy to programmatically work with the Chrome browser on the server. Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that. Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument. The username is the actual Instagram page that we want to scrape images from. Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram. Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall. So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page. Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot. Then you can use the page type method to select an element from the DOM and simulate typing characters into that form. If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument. From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in. So basically, you have unlimited possibilities for interacting with the website. Now after you've interacted with something, you might want to wait for the page to update. You can do this manually by saying wait for and then passing in a number of milliseconds to wait. Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM. Now if we want to actually scrape some data at any given time, We can use page evaluate. And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer. In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML. The code you write inside the evaluate callback is like being inside the actual browser. So you can use native browser APIs like document, query selector, all to grab all the images. And we can just map them down to the image source as an array of strings and return that from the evaluate call back. At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client. And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app. Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process. So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web. I'm I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip. Io to get access to even more content. Thanks for watching, and I will talk to you soon.
Today's
today's
video
video
is
is
all
all
about
about
extracting
extracting
data
data
from
from
the
the
Web.
web
In
in
a
a
perfect
perfect
world,
world
every
every
website
website
provides
provides
an
an
easy
easy
to
to
use
use
API
api
that
that
provides
provides
access
access
to
to
the
the
data
data
that
that
you
you
want.
want
But
but
unfortunately,
unfortunately
we
we
don't
don't
live
live
in
in
a
a
perfect
perfect
world.
world
But
but
as
as
hackers,
hackers
we
we
can
can
resort
resort
to
to
a
a
technique
technique
known
known
as
as
web
web
scraping
scraping
to
to
take
take
the
the
data
data
that
that
we
we
need
need
by
by
brute
brute
force.
force
And
and
I
i
know
know
that
that
sounds
sounds
kind
kind
of
of
shady,
shady
but
but
there
there
are
are
have
have
plenty
plenty
of
of
valid
valid
use
use
cases
cases
for
for
web
web
scraping
scraping
as
as
we'll
we'll
see
see
in
in
this
this
video.
video
For
for
example,
example
we'll
we'll
build
build
a
a
Node.
node
Js
js
Cloud
cloud
function
function
that
that
builds
builds
a
a
link
link
preview
preview
in
in
a
a
frontend
frontend
application,
application
then
then
we'll
we'll
move
move
on
on
to
to
a
a
more
more
questionable
questionable
use
use
case
case
where
where
we
we
actually
actually
log
log
into
into
an
an
Instagram
instagram
account
account
and
and
interact
interact
with
with
the
the
app
app
as
as
if
if
we
we
were
were
a
a
real
real
logged
logged
in
in
user.
user
If
if
you're
you're
new
new
here,
here
like
like
and
and
subscribe,
subscribe
and
and
congrats
congrats
to
to
Alex
alex
Karpov,
karpov
you're
you're
the
the
t
t
shirt
shirt
winner
winner
from
from
last
last
week's
week's
video.
video
But
but
there's
there's
another
another
one
one
going
going
out
out
with
with
this
this
video,
video
all
all
you
you
have
have
to
to
do
do
is
is
leave
leave
a
a
comment
comment
below.
below
Now,
now
in
in
this
this
video,
video
we're
we're
going
going
to
to
look
look
at
at
do
do
very
very
different
different
strategies
strategies
for
for
web
web
scraping
scraping
using
using
JavaScript
javascript
and
and
Node.
node
Js.
js
The
the
first
first
use
use
case
case
that
that
we'll
we'll
look
look
at
at
is
is
building
building
Link
link
Previews
previews
in
in
a
a
front
front
end
end
application.
application
You
you
see
see
this
this
in
in
applications
applications
like
like
Twitter
twitter
and
and
Slack
slack
where
where
the
the
user
user
includes
includes
a
a
URL
url
in
in
the
the
post,
post
and
and
then
then
it
it
will
will
automatically
automatically
generate
generate
a
a
preview
preview
for
for
that
that
URL.
url
We
we
can
can
implement
implement
this
this
feature
feature
from
from
scratch
scratch
using
using
a
a
Node.
node
Js
js
cloud
cloud
function.
function
And
and
the
the
reason
reason
we
we
need
need
to
to
do
do
this
this
with
with
Node.
node
Js
js
and
and
not
not
do
do
it
it
entirely
entirely
from
from
the
the
front
front
end
end
is
is
because
because
the
the
browser
browser
will
will
block
block
cross
cross
origin
origin
requests.
requests
So
so
if
if
your
your
website
website
has
has
a
a
script
script
that
that
tries
tries
to
to
make
make
a
a
request
request
to
to
a
a
different
different
website,
website
you'll
you'll
end
end
up
up
with
with
this
this
CORS
cors
error
error
in
in
the
the
browser.
browser
One
one
of
of
the
the
main
main
reasons
reasons
that
that
browsers
browsers
block
block
cross
cross
origin
origin
requests
requests
is
is
because
because
of
of
the
the
Spectre
spectre
security
security
vulnerability.
vulnerability
It's
it's
beyond
beyond
the
the
scope
scope
of
of
this
this
video,
video
but
but
basically
basically
making
making
a
a
client
client
side
side
request
request
to
to
a
a
third
third
party
party
website
website
could
could
allow
allow
an
an
attacker
attacker
to
to
read
read
what's
what's
in
in
memory
memory
on
on
the
the
client's
client's
device.
device
But
but
we
we
can
can
easily
easily
overcome
overcome
this
this
limitation
limitation
by
by
creating
creating
a
a
Cloud
cloud
Function
function
that
that
will
will
serve
serve
as
as
a
a
middleman.
middleman
On
on
our
our
front
front
end
end
application,
application
the
the
user
user
might
might
paste
paste
a
a
couple
couple
of
of
URLs
urls
into
into
a
a
form,
form
then
then
we'll
we'll
send
send
those
those
URLs
urls
to
to
our
our
Cloud
cloud
Function,
function
which
which
will
will
then
then
make
make
the
the
request
request
to
to
those
those
3rd
3rd
party
party
websites
websites
in
in
an
an
isolated
isolated
environment.
environment
And
and
this
this
is
is
where
where
the
the
web
web
scraping
scraping
comes
comes
in.
in
Because
because
the
the
Cloud
cloud
Function
function
is
is
going
going
to
to
retrieve
retrieve
an
an
HTML
html
document,
document
which
which
is
is
just
just
a
a
big
big
string
string
of
of
HTML,
html
then
then
it
it
will
will
process
process
that
that
HTML
html
looking
looking
for
for
meta
meta
tags
tags
that
that
we
we
can
can
use
use
for
for
the
the
link
link
preview
preview
things
things
like
like
the
the
description,
description
title,
title
and
and
featured
featured
image.
image
So
so
let's
let's
go
go
ahead
ahead
and
and
jump
jump
into
into
the
the
code,
code
and
and
the
the
first
first
thing
thing
we'll
we'll
do
do
is
is
initialize
initialize
Cloud
cloud
Functions
functions
in
in
this
this
project.
project
Now
now
we'll
we'll
mostly
mostly
be
be
focused
focused
on
on
back
back
end
end
code,
code
but
but
this
this
video
video
is
is
part
part
of
of
the
the
Minimum
minimum
Viable
viable
Product
product
series,
series
which
which
means
means
you
you
can
can
find
find
integrations
integrations
for
for
Flutter,
flutter
React,
react
and
and
Ionic
ionic
Angular
angular
on
on
Fireship
fireship
IO.
io
After
after
initializing
initializing
Firebase
firebase
functions,
functions
you'll
you'll
notice
notice
that
that
we
we
have
have
a
a
few
few
dependencies
dependencies
in
in
our
our
package.
package
Json.
json
First,
first
we
we
have
have
Cheerio,
cheerio
which
which
is
is
a
a
Node.
node
Js
js
implementation
implementation
of
of
jQuery,
jquery
which
which
just
just
makes
makes
it
it
a
a
lot
lot
easier
easier
to
to
scrape
scrape
web
web
pages.
pages
Then
then
we
we
have
have
CORS,
cors
so
so
we
we
can
can
specify
specify
which
which
URLs
urls
have
have
access
access
to
to
this
this
function.
function
Get
get
URLs
urls
is
is
a
a
utility
utility
for
for
extracting
extracting
URLs
urls
from
from
a
a
string.
string
And
and
Node
node
Fetch
fetch
is
is
a
a
Promise
promise
based
based
HTTP
http
library
library
that
that
mirrors
mirrors
the
the
Fetch
fetch
API
api
in
in
the
the
browser.
browser
And
and
lastly,
lastly
we
we
have
have
Puppeteer,
puppeteer
which
which
is
is
basically
basically
a
a
version
version
of
of
Chrome
chrome
that
that
runs
runs
in
in
Node.
node
Js,
js
but
but
we
we
won't
won't
get
get
into
into
that
that
one
one
until
until
we
we
get
get
to
to
the
the
second
second
function
function
to
to
scrape
scrape
Instagram.
instagram
Now
now
if
if
we
we
go
go
to
to
the
the
function's
function's
code,
code
we'll
we'll
import
import
CORS
cors
and
and
set
set
origin
origin
to
to
true,
true
which
which
will
will
give
give
any
any
URL
url
access
access
to
to
this
this
function
function
endpoint.
endpoint
And
and
CORS
cors
is
is
only
only
required
required
if
if
you're
you're
building
building
an
an
HTTP
http
function.
function
If
if
you're
you're
using
using
a
a
Firestore
firestore
function
function
or
or
a
a
callable
callable
function,
function
you
you
can
can
just
just
keep
keep
it
it
altogether.
altogether
And
and
this
this
code
code
should
should
be
be
very
very
easy
easy
to
to
modify
modify
because
because
what
what
we're
we're
doing
doing
is
is
creating
creating
a
a
helper
helper
function
function
called
called
Scrape
scrape
Metatags,
metatags
and
and
it
it
simply
simply
takes
takes
some
some
text
text
as
as
an
an
argument
argument
which
which
you
you
might
might
think
think
of
of
as
as
a
a
tweet
tweet
that
that
has
has
a
a
couple
couple
of
of
URLs
urls
included
included
within
within
it.
it
The
the
first
first
thing
thing
we
we
wanna
wanna
do
do
with
with
this
this
text
text
is
is
extract
extract
all
all
of
of
the
the
valid
valid
URLs
urls
from
from
it,
it
which
which
we
we
can
can
do
do
with
with
the
the
get
get
URLs
urls
utility.
utility
Then
then
what
what
we'll
we'll
do
do
is
is
map
map
all
all
of
of
these
these
URLs
urls
into
into
individual
individual
requests
requests
and
and
then
then
run
run
them
them
concurrently
concurrently
with
with
Promise
promise
dot
dot
all.
all
Now
now
a
a
key
key
thing
thing
to
to
keep
keep
in
in
mind
mind
with
with
this
this
strategy
strategy
is
is
that
that
it
it
will
will
not
not
execute
execute
any
any
JavaScript
javascript
on
on
the
the
website
website
that
that
it
it
requests.
requests
And
and
this
this
is
is
pretty
pretty
important
important
because
because
a
a
lot
lot
of
of
modern
modern
web
web
apps
apps
like
like
Instagram,
instagram
for
for
example,
example
use
use
things
things
like
like
React,
react
and
and
so
so
the
the
content
content
isn't
isn't
rendered
rendered
until
until
the
the
JavaScript
javascript
is
is
executed.
executed
So
so
this
this
first
first
strategy
strategy
with
with
NodeFetch
nodefetch
and
and
Cheerio
cheerio
is
is
very
very
fast
fast
and
and
efficient,
efficient
but
but
it
it
doesn't
doesn't
get
get
the
the
dynamic
dynamic
content
content
that's
that's
rendered
rendered
after
after
the
the
fact
fact
by
by
JavaScript.
javascript
And
and
that's
that's
perfectly
perfectly
fine
fine
for
for
a
a
link
link
preview
preview
because
because
the
the
meta
meta
tags
tags
on
on
a
a
URL
url
are
are
typically
typically
pre
pre
rendered.
rendered
Now
now
for
for
each
each
URL,
url
we'll
we'll
make
make
a
a
request
request
using
using
fetch,
fetch
and
and
then
then
we'll
we'll
convert
convert
the
the
response
response
to
to
text,
text
which
which
will
will
just
just
be
be
a
a
string
string
of
of
HTML.
html
From
from
there,
there
we'll
we'll
load
load
that
that
text
text
into
into
Cheerio,
cheerio
which
which
makes
makes
it
it
really
really
easy
easy
to
to
parse
parse
nodes
nodes
of
of
the
the
Dom.
dom
If
if
you're
you're
not
not
familiar
familiar
with
with
jQuery,
jquery
the
the
only
only
thing
thing
you
you
really
really
need
need
to
to
know
know
for
for
this
this
video
video
is
is
that
that
need
need
dollar
dollar
sign
sign
is
is
a
a
function
function
that
that
allows
allows
you
you
to
to
select
select
notes
notes
from
from
the
the
DOM
dom
with
with
a
a
selector.
selector
So
so
the
the
first
first
thing
thing
that
that
we'll
we'll
scrape
scrape
from
from
this
this
HTML
html
is
is
the
the
title,
title
which
which
we
we
can
can
do
do
easily
easily
with
with
dollar
dollar
sign
sign
title
title
and
and
then
then
grab
grab
the
the
text.
text
And
and
then
then
we
we
might
might
also
also
want
want
to
to
grab
grab
the
the
website's
website's
favicon,
favicon
and
and
then
then
we
we
can
can
grab
grab
the
the
href
href
attribute
attribute
that
that
will
will
give
give
us
us
the
the
link
link
to
to
the
the
actual
actual
image
image
so
so
we
we
can
can
show
show
that
that
in
in
the
the
link
link
preview.
preview
But
but
most
most
of
of
the
the
data
data
that
that
we
we
want
want
will
will
come
come
from
from
meta
meta
tags.
tags
For
for
example,
example
to
to
show
show
the
the
site
site
description,
description
will
will
grab
grab
the
the
description
description
meta
meta
tag
tag
and
and
then
then
grab
grab
the
the
content
content
attribute.
attribute
But
but
the
the
thing
thing
about
about
meta
meta
tags
tags
is
is
that
that
there
there
might
might
be
be
multiple
multiple
formats
formats
included
included
on
on
a
a
single
single
website.
website
Developers
developers
might
might
include
include
tags
tags
for
for
Open
open
Graph
graph
or
or
Twitter.
twitter
And
and
in
in
our
our
case,
case
we
we
don't
don't
really
really
care
care
what
what
the
the
source
source
is,
is
we
we
just
just
want
want
to
to
get
get
the
the
right
right
data
data
to
to
show
show
in
in
the
the
UI
ui
for
for
the
the
link
link
preview.
preview
So
so
so
so
we'll
we'll
go
go
ahead
ahead
and
and
create
create
a
a
new
new
function
function
inside
inside
of
of
this
this
function,
function
so
so
it
it
has
has
access
access
to
to
the
the
dollar
dollar
sign,
sign
and
and
its
its
job
job
is
is
to
to
just
just
look
look
for
for
the
the
first
first
available
available
meta
meta
tag.
tag
You'll
you'll
likely
likely
want
want
to
to
scrape
scrape
as
as
many
many
meta
meta
tags
tags
as
as
possible,
possible
so
so
creating
creating
this
this
little
little
helper
helper
function
function
will
will
keep
keep
your
your
code
code
dry
dry
and
and
concise.
concise
So
so
instead
instead
of
of
reimplementing
reimplementing
this
this
logic
logic
over
over
and
and
over
over
again,
again
we
we
just
just
call
call
get
get
meta
meta
tags
tags
for
for
the
the
tags
tags
that
that
we
we
wanna
wanna
return
return
back
back
to
to
the
the
front
front
end
end
for
for
the
the
link
link
preview.
preview
Now
now
that
that
takes
takes
care
care
of
of
the
the
logic
logic
for
for
scraping
scraping
the
the
meta
meta
tags,
tags
but
but
to
to
actually
actually
test
test
this
this
out,
out
we'll
we'll
want
want
to
to
set
set
it
it
up
up
in
in
a
a
Firebase
firebase
Cloud
cloud
function.
function
I'm
i'm
setting
setting
it
it
up
up
as
as
an
an
HTTP
http
function,
function
but
but
feel
feel
free
free
to
to
modify
modify
it
it
to
to
work
work
with
with
Firestore
firestore
or
or
any
any
other
other
Cloud
cloud
Function
function
trigger.
trigger
We'll
we'll
go
go
ahead
ahead
and
and
call
call
this
this
function
function
scraper,
scraper
and
and
then
then
we'll
we'll
wrap
wrap
it
it
in
in
CORS
cors
to
to
ensure
ensure
that
that
it
it
can
can
be
be
called
called
successfully
successfully
from
from
any
any
front
front
end
end
application.
application
The
the
front
front
end
end
application
application
will
will
send
send
a
a
JSON
json
string
string
as
as
the
the
request
request
body.
body
So
so
we'll
we'll
go
go
ahead
ahead
and
and
parse
parse
that
that
string,
string
and
and
then
then
we'll
we'll
pass
pass
in
in
the
the
body
body
text
text
to
to
our
our
scrape
scrape
meta
meta
tags
tags
function.
function
The
the
result
result
of
of
that
that
function
function
will
will
be
be
an
an
array
array
of
of
objects
objects
where
where
each
each
object
object
is
is
the
the
URL
url
data
data
that
that
was
was
scrape
scrape
from
from
that
that
3rd
3rd
party
party
website.
website
And
and
with
with
that,
that
we
we
can
can
run
run
Firebase
firebase
serve,
serve
and
and
now
now
our
our
Cloud
cloud
Function
function
is
is
ready
ready
to
to
test
test
locally.
locally
I'm
i'm
using
using
an
an
application
application
called
called
Insomnia
insomnia
to
to
test
test
the
the
endpoint,
endpoint
But
but
make
make
sure
sure
to
to
check
check
out
out
the
the
front
front
end
end
integration
integration
videos
videos
on
on
Fireship
fireship
IO
io
for
for
React,
react
Angular,
angular
and
and
Flutter.
flutter
As
as
you
you
can
can
see
see
here,
here
when
when
we
we
send
send
a
a
request
request
with
with
some
some
text
text
that
that
resembles
resembles
a
a
tweet,
tweet
We
we
get
get
a
a
response
response
back
back
that
that
is
is
an
an
array
array
of
of
objects
objects
with
with
the
the
link
link
data
data
from
from
the
the
meta
meta
tags
tags
on
on
those
those
3rd
3rd
party
party
websites.
websites
And
and
then
then
you
you
can
can
use
use
this
this
data
data
in
in
your
your
frontend
frontend
application
application
display
display
a
a
component
component
or
or
widget
widget
with
with
the
the
actual
actual
UI
ui
for
for
the
the
link
link
preview.
preview
Now
now
let's
let's
take
take
a
a
second
second
to
to
recap
recap
the
the
pros
pros
and
and
cons
cons
of
of
the
the
code
code
we
we
just
just
implemented.
implemented
Cheerio
cheerio
along
along
with
with
Node.
node
Fetch
fetch
make
make
it
it
easy
easy
to
to
parse
parse
data
data
from
from
a
a
pre
pre
rendered
rendered
or
or
server
server
rendered
rendered
HTML
html
page.
page
But
but
like
like
I
i
mentioned
mentioned
earlier,
earlier
a
a
lot
lot
of
of
modern
modern
sites
sites
use
use
JavaScript
javascript
to
to
render
render
content
content
after
after
the
the
initial
initial
page
page
load.
load
If
if
you
you
go
go
to
to
any
any
website,
website
you
you
can
can
see
see
the
the
initially
initially
rendered
rendered
content
content
by
by
viewing
viewing
the
the
page
page
source.
source
If
if
you
you
go
go
to
to
a
a
web
web
application
application
like
like
Instagram,
instagram
which
which
is
is
built
built
with
with
React,
react
you'll
you'll
notice
notice
that
that
the
the
initial
initial
source
source
code
code
doesn't
doesn't
contain
contain
any
any
of
of
the
the
image
image
URLs
urls
that
that
you
you
see
see
on
on
the
the
screen.
screen
That's
that's
because
because
the
the
initial
initial
page
page
load
load
is
is
just
just
a
a
shell
shell
of
of
content
content
and
and
JavaScript
javascript
loads
loads
everything
everything
else
else
dynamically
dynamically
after
after
the
the
fact.
fact
And
and
this
this
initial
initial
source
source
code
code
differs
differs
from
from
the
the
elements
elements
tab
tab
on
on
the
the
browser
browser
console,
console
don't,
don't
which
which
will
will
reflect
reflect
the
the
changes
changes
made
made
to
to
the
the
DOM
dom
by
by
JavaScript.
javascript
Now
now
we
we
can
can
still
still
scrape
scrape
this
this
content,
content
but
but
we
we
need
need
to
to
have
have
a
a
browser
browser
environment
environment
that
that
will
will
execute
execute
JavaScript.
javascript
And
and
a
a
great
great
tool
tool
for
for
doing
doing
that
that
is
is
Puppeteer.
puppeteer
Puppeteer
puppeteer
makes
makes
it
it
easy
easy
to
to
programmatically
programmatically
work
work
with
with
the
the
Chrome
chrome
browser
browser
on
on
the
the
server.
server
Not
not
only
only
does
does
it
it
execute
execute
JavaScript,
javascript
but
but
it
it
also
also
allows
allows
you
you
to
to
emulate
emulate
a
a
real
real
user
user
by
by
clicking
clicking
on
on
buttons
buttons
and
and
filling
filling
out
out
forms
forms
and
and
things
things
like
like
that.
that
Now
now
getting
getting
back
back
to
to
our
our
JavaScript
javascript
code,
code
we'll
we'll
go
go
ahead
ahead
and
and
import
import
puppet
puppet
here
here
and
and
then
then
create
create
a
a
new
new
async
async
function
function
called
called
scrape
scrape
images
images
that
that
takes
takes
a
a
username
username
as
as
an
an
argument.
argument
The
the
username
username
is
is
the
the
actual
actual
Instagram
instagram
page
page
that
that
we
we
want
want
to
to
scrape
scrape
images
images
from.
from
Now
now
before
before
we
we
go
go
any
any
further,
further
I
i
need
need
to
to
point
point
out
out
that
that
this
this
is
is
very
very
likely
likely
violating
violating
the
the
terms
terms
of
of
service
service
with
with
Instagram.
instagram
Web
web
scraping
scraping
in
in
general
general
falls
falls
into
into
a
a
gray
gray
area,
area
and
and
most
most
apps
apps
likely
likely
aren't
aren't
going
going
to
to
allow
allow
it,
it
especially
especially
if
if
the
the
content
content
is
is
behind
behind
an
an
authentication
authentication
wall
wall
or
or
even
even
worse,
worse
a
a
paywall.
paywall
So
so
now
now
that
that
you
you
know
know
that,
that
let's
let's
go
go
ahead
ahead
and
and
initialize
initialize
Puppeteer,
puppeteer
and
and
then
then
we'll
we'll
point
point
it
it
to
to
the
the
Instagram
instagram
login
login
page.
page
Going
going
to
to
this
this
URL
url
with
with
Puppeteer
puppeteer
is
is
just
just
like
like
visiting
visiting
it
it
in
in
a
a
regular
regular
Chrome
chrome
browser,
browser
and
and
you
you
can
can
inspect
inspect
the
the
state
state
of
of
the
the
website
website
at
at
any
any
time
time
by
by
simply
simply
taking
taking
a
a
screenshot.
screenshot
Then
then
you
you
can
can
use
use
the
the
page
page
type
type
method
method
to
to
select
select
an
an
element
element
from
from
the
the
DOM
dom
and
and
simulate
simulate
typing
typing
characters
characters
into
into
that
that
form.
form
If
if
we
we
look
look
at
at
the
the
Instagram
instagram
login
login
form,
form
you
you
can
can
see
see
an
an
input
input
that
that
has
has
a
a
name
name
of
of
username,
username
so
so
we
we
can
can
use
use
that
that
selector
selector
and
and
then
then
type
type
in
in
our
our
username
username
as
as
the
the
second
second
argument.
argument
From
from
there,
there
we'll
we'll
grab
grab
the
the
password
password
select
select
her
her
and
and
type
type
in
in
the
the
password,
password
and
and
lastly,
lastly
click
click
on
on
the
the
submit
submit
button
button
to
to
submit
submit
the
the
form
form
and
and
log
log
the
the
user
user
in.
in
So
so
basically,
basically
you
you
have
have
unlimited
unlimited
possibilities
possibilities
for
for
interacting
interacting
with
with
the
the
website.
website
Now
now
after
after
you've
you've
interacted
interacted
with
with
something,
something
you
you
might
might
want
want
to
to
wait
wait
for
for
the
the
page
page
to
to
update.
update
You
you
can
can
do
do
this
this
manually
manually
by
by
saying
saying
wait
wait
for
for
and
and
then
then
passing
passing
in
in
a
a
number
number
of
of
milliseconds
milliseconds
to
to
wait.
wait
Or
or
in
in
our
our
case,
case
we
we
know
know
that
that
we're
we're
waiting
waiting
for
for
React
react
to
to
render
render
the
the
image
image
tags,
tags
so
so
we'll
we'll
just
just
go
go
ahead
ahead
and
and
wait
wait
for
for
that
that
selector
selector
to
to
appear
appear
in
in
the
the
DOM.
dom
Now
now
if
if
we
we
want
want
to
to
actually
actually
scrape
scrape
some
some
data
data
at
at
any
any
given
given
time,
time
We
we
can
can
use
use
page
page
evaluate.
evaluate
And
and
this
this
gives
gives
us
us
a
a
callback
callback
function,
function
which
which
gives
gives
us
us
access
access
to
to
the
the
DOM
dom
APIs
apis
in
in
Puppeteer.
puppeteer
In
in
other
other
words,
words
you
you
can
can
pause
pause
the
the
execution
execution
of
of
the
the
browser
browser
at
at
any
any
given
given
time
time
and
and
scrape
scrape
whatever
whatever
data
data
you
you
want
want
from
from
the
the
HTML.
html
The
the
code
code
you
you
write
write
inside
inside
the
the
evaluate
evaluate
callback
callback
is
is
like
like
being
being
inside
inside
the
the
actual
actual
browser.
browser
So
so
you
you
can
can
use
use
native
native
browser
browser
APIs
apis
like
like
document,
document
query
query
selector,
selector
all
all
to
to
grab
grab
all
all
the
the
images.
images
And
and
we
we
can
can
just
just
map
map
them
them
down
down
to
to
the
the
image
image
source
source
as
as
an
an
array
array
of
of
strings
strings
and
and
return
return
that
that
from
from
the
the
evaluate
evaluate
call
call
back.
back
At
at
this
this
point,
point
you
you
could
could
continue
continue
interacting
interacting
with
with
the
the
browser
browser
and
and
scrape
scrape
data
data
from
from
additional
additional
frames,
frames
but
but
in
in
our
our
case,
case
we'll
we'll
be
be
done
done
here
here
and
and
just
just
call
call
browser
browser
close
close
and
and
and
and
then
then
return
return
this
this
data
data
back
back
to
to
the
the
client.
client
And
and
if
if
we
we
go
go
ahead
ahead
and
and
make
make
the
the
request
request
from
from
Insomnia,
insomnia
you'll
you'll
see
see
we
we
get
get
an
an
array
array
of
of
URLs
urls
back
back
even
even
know
know
all
all
the
the
URLs
urls
are
are
generated
generated
dynamically
dynamically
in
in
the
the
front
front
end
end
Instagram
instagram
React
react
app.
app
Now
now
you
you
may
may
have
have
noticed
noticed
it
it
took
took
a
a
lot
lot
longer
longer
to
to
process
process
that
that
request,
request
but
but
keep
keep
in
in
mind
mind
find
find
that
that
we
we
actually
actually
logged
logged
into
into
an
an
Instagram
instagram
account
account
and
and
had
had
multiple
multiple
delays
delays
in
in
the
the
process.
process
So
so
Puppeteer
puppeteer
offers
offers
all
all
kinds
kinds
of
of
possibilities
possibilities
for
for
extracting
extracting
dynamic
dynamic
data
data
from
from
the
the
web.
web
I'm
i'm
I'm
i'm
gonna
gonna
go
go
ahead
ahead
and
and
wrap
wrap
things
things
up
up
there.
there
If
if
this
this
video
video
helped
helped
you,
you
please
please
like
like
and
and
subscribe,
subscribe
and
and
consider
consider
becoming
becoming
a
a
pro
pro
member
member
at
at
FireShip.
fireship
Io
io
to
to
get
get
access
access
to
to
even
even
more
more
content.
content
Thanks
thanks
for
for
watching,
watching
and
and
I
i
will
will
talk
talk
to
to
you
you
soon.
soon
success
The speakers discuss various strategies for scraping data from the web, including building link previews in a front end application, using a browser blocker for cross- origin requests, and creating a Cloud Function to handle requests. They emphasize the efficiency and efficiency of these strategies, but also discuss the need for dynamic content creation and the use of a variable in the URL. The process of scraping web pages is discussed, including using a trigger to test the endpoint and importing and creating new functions. The speakers also explain how to use the function scraper in the front end of a web application, and how to use the Cloud functions trigger to test the endpoint and run a web app to test the endpoint. They suggest becoming a pro member at Fire Mm if the process takes longer than expected.
A Guide to Web Scraping with Node.js
Today's video is all about extracting data from the Web. In a perfect world, every website provides an easy to use API that provides access to the data that you want. But unfortunately, we don't live in a perfect world. But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force. And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video. For example, we'll build a Node. Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user. If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video. But there's another one going out with this video, all you have to do is leave a comment below. Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node. Js. The first use case that we'll look at is building Link Previews in a front end application. You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL. We can implement this feature from scratch using a Node. Js cloud function. And the reason we need to do this with Node. Js and not do it entirely from the front end is because the browser will block cross origin requests. So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser. One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability. It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device. But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman. On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment. And this is where the web scraping comes in. Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image. So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project. Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO. After initializing Firebase functions, you'll notice that we have a few dependencies in our package. Json. First, we have Cheerio, which is a Node. Js implementation of jQuery, which just makes it a lot easier to scrape web pages. Then we have CORS, so we can specify which URLs have access to this function. Get URLs is a utility for extracting URLs from a string. And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser. And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node. Js, but we won't get into that one until we get to the second function to scrape Instagram. Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint. And CORS is only required if you're building an HTTP function. If you're using a Firestore function or a callable function, you can just keep it altogether. And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it. The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility. Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all. Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests. And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed. So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript. And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered. Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML. From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom. If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector. So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text. And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview. But most of the data that we want will come from meta tags. For example, to show the site description, will grab the description meta tag and then grab the content attribute. But the thing about meta tags is that there might be multiple formats included on a single website. Developers might include tags for Open Graph or Twitter. And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview. So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag. You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise. So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview. Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function. I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger. We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application. The front end application will send a JSON string as the request body. So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function. The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website. And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally. I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter. As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites. And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview. Now let's take a second to recap the pros and cons of the code we just implemented. Cheerio along with Node. Fetch make it easy to parse data from a pre rendered or server rendered HTML page. But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load. If you go to any website, you can see the initially rendered content by viewing the page source. If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen. That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact. And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript. Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript. And a great tool for doing that is Puppeteer. Puppeteer makes it easy to programmatically work with the Chrome browser on the server. Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that. Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument. The username is the actual Instagram page that we want to scrape images from. Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram. Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall. So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page. Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot. Then you can use the page type method to select an element from the DOM and simulate typing characters into that form. If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument. From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in. So basically, you have unlimited possibilities for interacting with the website. Now after you've interacted with something, you might want to wait for the page to update. You can do this manually by saying wait for and then passing in a number of milliseconds to wait. Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM. Now if we want to actually scrape some data at any given time, We can use page evaluate. And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer. In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML. The code you write inside the evaluate callback is like being inside the actual browser. So you can use native browser APIs like document, query selector, all to grab all the images. And we can just map them down to the image source as an array of strings and return that from the evaluate call back. At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client. And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app. Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process. So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web. I'm I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip. Io to get access to even more content. Thanks for watching, and I will talk to you soon.
The speakers discuss various strategies for scraping data from the web, including building link previews in a front end application, using a browser blocker for cross- origin requests, and creating a Cloud Function to handle requests. They emphasize the efficiency and efficiency of these strategies, but also discuss the need for dynamic content creation and the use of a variable in the URL. The process of scraping web pages is discussed, including using a trigger to test the endpoint and importing and creating new functions. The speakers also explain how to use the function scraper in the front end of a web application, and how to use the Cloud functions trigger to test the endpoint and run a web app to test the endpoint. They suggest becoming a pro member at Fire Mm if the process takes longer than expected.

        [INST]
        You are youtuber called Fireship you make engaging high-intensity and entertaining coding tutorials and tech news. 
        you covers a wide range of topics relevant to programmers, aiming to help them learn and improve their skills quickly.
        
        Given the title of the video : A Guide to Web Scraping with Node.js 
        and a small summary : The speakers discuss various strategies for scraping data from the web, including building link previews in a front end application, using a browser blocker for cross- origin requests, and creating a Cloud Function to handle requests. They emphasize the efficiency and efficiency of these strategies, but also discuss the need for dynamic content creation and the use of a variable in the URL. The process of scraping web pages is discussed, including using a trigger to test the endpoint and importing and creating new functions. The speakers also explain how to use the function scraper in the front end of a web application, and how to use the Cloud functions trigger to test the endpoint and run a web app to test the endpoint. They suggest becoming a pro member at Fire Mm if the process takes longer than expected.
        [/INST]
        
        Generate the video : Today's video is all about extracting data from the Web. In a perfect world, every website provides an easy to use API that provides access to the data that you want. But unfortunately, we don't live in a perfect world. But as hackers, we can resort to a technique known as web scraping to take the data that we need by brute force. And I know that sounds kind of shady, but there are have plenty of valid use cases for web scraping as we'll see in this video. For example, we'll build a Node. Js Cloud function that builds a link preview in a frontend application, then we'll move on to a more questionable use case where we actually log into an Instagram account and interact with the app as if we were a real logged in user. If you're new here, like and subscribe, and congrats to Alex Karpov, you're the t shirt winner from last week's video. But there's another one going out with this video, all you have to do is leave a comment below. Now, in this video, we're going to look at do very different strategies for web scraping using JavaScript and Node. Js. The first use case that we'll look at is building Link Previews in a front end application. You see this in applications like Twitter and Slack where the user includes a URL in the post, and then it will automatically generate a preview for that URL. We can implement this feature from scratch using a Node. Js cloud function. And the reason we need to do this with Node. Js and not do it entirely from the front end is because the browser will block cross origin requests. So if your website has a script that tries to make a request to a different website, you'll end up with this CORS error in the browser. One of the main reasons that browsers block cross origin requests is because of the Spectre security vulnerability. It's beyond the scope of this video, but basically making a client side request to a third party website could allow an attacker to read what's in memory on the client's device. But we can easily overcome this limitation by creating a Cloud Function that will serve as a middleman. On our front end application, the user might paste a couple of URLs into a form, then we'll send those URLs to our Cloud Function, which will then make the request to those 3rd party websites in an isolated environment. And this is where the web scraping comes in. Because the Cloud Function is going to retrieve an HTML document, which is just a big string of HTML, then it will process that HTML looking for meta tags that we can use for the link preview things like the description, title, and featured image. So let's go ahead and jump into the code, and the first thing we'll do is initialize Cloud Functions in this project. Now we'll mostly be focused on back end code, but this video is part of the Minimum Viable Product series, which means you can find integrations for Flutter, React, and Ionic Angular on Fireship IO. After initializing Firebase functions, you'll notice that we have a few dependencies in our package. Json. First, we have Cheerio, which is a Node. Js implementation of jQuery, which just makes it a lot easier to scrape web pages. Then we have CORS, so we can specify which URLs have access to this function. Get URLs is a utility for extracting URLs from a string. And Node Fetch is a Promise based HTTP library that mirrors the Fetch API in the browser. And lastly, we have Puppeteer, which is basically a version of Chrome that runs in Node. Js, but we won't get into that one until we get to the second function to scrape Instagram. Now if we go to the function's code, we'll import CORS and set origin to true, which will give any URL access to this function endpoint. And CORS is only required if you're building an HTTP function. If you're using a Firestore function or a callable function, you can just keep it altogether. And this code should be very easy to modify because what we're doing is creating a helper function called Scrape Metatags, and it simply takes some text as an argument which you might think of as a tweet that has a couple of URLs included within it. The first thing we wanna do with this text is extract all of the valid URLs from it, which we can do with the get URLs utility. Then what we'll do is map all of these URLs into individual requests and then run them concurrently with Promise dot all. Now a key thing to keep in mind with this strategy is that it will not execute any JavaScript on the website that it requests. And this is pretty important because a lot of modern web apps like Instagram, for example, use things like React, and so the content isn't rendered until the JavaScript is executed. So this first strategy with NodeFetch and Cheerio is very fast and efficient, but it doesn't get the dynamic content that's rendered after the fact by JavaScript. And that's perfectly fine for a link preview because the meta tags on a URL are typically pre rendered. Now for each URL, we'll make a request using fetch, and then we'll convert the response to text, which will just be a string of HTML. From there, we'll load that text into Cheerio, which makes it really easy to parse nodes of the Dom. If you're not familiar with jQuery, the only thing you really need to know for this video is that need dollar sign is a function that allows you to select notes from the DOM with a selector. So the first thing that we'll scrape from this HTML is the title, which we can do easily with dollar sign title and then grab the text. And then we might also want to grab the website's favicon, and then we can grab the href attribute that will give us the link to the actual image so we can show that in the link preview. But most of the data that we want will come from meta tags. For example, to show the site description, will grab the description meta tag and then grab the content attribute. But the thing about meta tags is that there might be multiple formats included on a single website. Developers might include tags for Open Graph or Twitter. And in our case, we don't really care what the source is, we just want to get the right data to show in the UI for the link preview. So so we'll go ahead and create a new function inside of this function, so it has access to the dollar sign, and its job is to just look for the first available meta tag. You'll likely want to scrape as many meta tags as possible, so creating this little helper function will keep your code dry and concise. So instead of reimplementing this logic over and over again, we just call get meta tags for the tags that we wanna return back to the front end for the link preview. Now that takes care of the logic for scraping the meta tags, but to actually test this out, we'll want to set it up in a Firebase Cloud function. I'm setting it up as an HTTP function, but feel free to modify it to work with Firestore or any other Cloud Function trigger. We'll go ahead and call this function scraper, and then we'll wrap it in CORS to ensure that it can be called successfully from any front end application. The front end application will send a JSON string as the request body. So we'll go ahead and parse that string, and then we'll pass in the body text to our scrape meta tags function. The result of that function will be an array of objects where each object is the URL data that was scrape from that 3rd party website. And with that, we can run Firebase serve, and now our Cloud Function is ready to test locally. I'm using an application called Insomnia to test the endpoint, But make sure to check out the front end integration videos on Fireship IO for React, Angular, and Flutter. As you can see here, when we send a request with some text that resembles a tweet, We get a response back that is an array of objects with the link data from the meta tags on those 3rd party websites. And then you can use this data in your frontend application display a component or widget with the actual UI for the link preview. Now let's take a second to recap the pros and cons of the code we just implemented. Cheerio along with Node. Fetch make it easy to parse data from a pre rendered or server rendered HTML page. But like I mentioned earlier, a lot of modern sites use JavaScript to render content after the initial page load. If you go to any website, you can see the initially rendered content by viewing the page source. If you go to a web application like Instagram, which is built with React, you'll notice that the initial source code doesn't contain any of the image URLs that you see on the screen. That's because the initial page load is just a shell of content and JavaScript loads everything else dynamically after the fact. And this initial source code differs from the elements tab on the browser console, don't, which will reflect the changes made to the DOM by JavaScript. Now we can still scrape this content, but we need to have a browser environment that will execute JavaScript. And a great tool for doing that is Puppeteer. Puppeteer makes it easy to programmatically work with the Chrome browser on the server. Not only does it execute JavaScript, but it also allows you to emulate a real user by clicking on buttons and filling out forms and things like that. Now getting back to our JavaScript code, we'll go ahead and import puppet here and then create a new async function called scrape images that takes a username as an argument. The username is the actual Instagram page that we want to scrape images from. Now before we go any further, I need to point out that this is very likely violating the terms of service with Instagram. Web scraping in general falls into a gray area, and most apps likely aren't going to allow it, especially if the content is behind an authentication wall or even worse, a paywall. So now that you know that, let's go ahead and initialize Puppeteer, and then we'll point it to the Instagram login page. Going to this URL with Puppeteer is just like visiting it in a regular Chrome browser, and you can inspect the state of the website at any time by simply taking a screenshot. Then you can use the page type method to select an element from the DOM and simulate typing characters into that form. If we look at the Instagram login form, you can see an input that has a name of username, so we can use that selector and then type in our username as the second argument. From there, we'll grab the password select her and type in the password, and lastly, click on the submit button to submit the form and log the user in. So basically, you have unlimited possibilities for interacting with the website. Now after you've interacted with something, you might want to wait for the page to update. You can do this manually by saying wait for and then passing in a number of milliseconds to wait. Or in our case, we know that we're waiting for React to render the image tags, so we'll just go ahead and wait for that selector to appear in the DOM. Now if we want to actually scrape some data at any given time, We can use page evaluate. And this gives us a callback function, which gives us access to the DOM APIs in Puppeteer. In other words, you can pause the execution of the browser at any given time and scrape whatever data you want from the HTML. The code you write inside the evaluate callback is like being inside the actual browser. So you can use native browser APIs like document, query selector, all to grab all the images. And we can just map them down to the image source as an array of strings and return that from the evaluate call back. At this point, you could continue interacting with the browser and scrape data from additional frames, but in our case, we'll be done here and just call browser close and and then return this data back to the client. And if we go ahead and make the request from Insomnia, you'll see we get an array of URLs back even know all the URLs are generated dynamically in the front end Instagram React app. Now you may have noticed it took a lot longer to process that request, but keep in mind find that we actually logged into an Instagram account and had multiple delays in the process. So Puppeteer offers all kinds of possibilities for extracting dynamic data from the web. I'm I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe, and consider becoming a pro member at FireShip. Io to get access to even more content. Thanks for watching, and I will talk to you soon.
        