Learn how to use Puppeteer with Firebase Cloud Functions to perform serverside rendering of any frontend app like Ionic 4 or Angular https://angularfirebase.com/lessons/how-to-use-puppeteer-on-firebase-cloud-functions/

- Puppeteer https://github.com/GoogleChrome/puppeteer
- Rendertron Episode https://angularfirebase.com/lessons/seo-angular-part-1-rendertron-meta-tags/
- Functions Course https://projects.angularfirebase.com/p/firebase-cloud-functions-foundations
2024-02-04T07:21:05.694Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
d0e1b79f-fef0-48a0-892f-bc9b1fbbd32c
d53ca3943b4b9348404468d2bce0ce0769c6913469b2384b035e6bc235dd2ab0
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment.
Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function.
If you're new to the channel, like and subscribe.
And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course.
So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome.
Headless Chrome is just a web browser that can run on your server.
And Puppeteer is a high level API allowing you to interact with the browser.
It's very intuitive to use and you get direct access to the DOM on the rendered page.
So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well.
If we go back to episode 66, We used an app called Rendertron to server side render an Angular application.
But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container.
In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function.
I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework.
The Cloud Functions back end is completely agnostic to your front end.
If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option.
And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new.
From there, we'll CD into the functions directory and then we'll need to set up Node.
Js version 8 as the runtime.
Open the package JSON, create an object called engines, then set node to a string value of 8.
Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config.
This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer.
And the last piece of setup is to install Puppeteer with npm.
You'll notice that that downloads Chromium, which is over a 100 megabytes in size.
And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function.
So we're going to beef that up in the next step.
Let's go ahead and open up the index.
Ts file and then we'll import Puppeteer in addition to Firebase functions.
The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function.
But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory.
Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function.
The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch.
And then there's a number of different configuration options that you can pass into launch.
In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly.
So that's the equivalent of opening up a browser.
The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser.
And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods.
The next step is to navigate to an actual URL.
In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages.
When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits.
In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there.
At this point, there's all kinds of different things we can do.
We might wanna scrape content from the page.
We might want to inject Jquery and make manipulations to it.
We can take a screenshot and do pretty much anything else you can do in a regular web browser.
But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function.
Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application.
So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML.
So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter.
But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser.
Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website.
Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response.
So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world.
In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that.
Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting.
So we'll import the serialized function from Rendertron.
Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing.
Then we'll write a Cloud Function specifically for rendering pages with Puppeteer.
This first render function is going to look almost identical to the one that we just wrote.
But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process.
And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL.
And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function.
The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint.
Then what we want to do is determine whether or not the request is coming from a bot or a human.
If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page.
I have a list of common bots here.
And then we can see the bot on the request by looking at the User Agent header.
Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header.
So if the length is 0, then we know we're not dealing with a bot.
But if it's greater than 0, we know that we should render this page with Puppeteer.
If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron.
We can do that by calling the render function that we created previously that handles all the puppeteer stuff.
So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page.
Then we can take that HTML and send it as the response body from this function.
Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back.
A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account.
You can either deploy this function or run it locally.
Then at this point, I'm assuming that you already have a Firebase Hosting account set up.
And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file.
We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote.
Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app.
And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards.
So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface.
There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff.
I'm gonna go ahead and wrap it up there.
If you want to see more advanced puppeteer content, leave me a comment down below.
And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com.
You'll get unlimited access to all pro content as well as my full courses and a free copy of my book.
Thanks for watching and I'll talk to you soon.

Speaker 0: When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment. Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function. If you're new to the channel, like and subscribe. And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course. So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome.

Headless Chrome is just a web browser that can run on your server. And Puppeteer is a high level API allowing you to interact with the browser. It's very intuitive to use and you get direct access to the DOM on the rendered page. So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well. If we go back to episode 66, We used an app called Rendertron to server side render an Angular application.

But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container. In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function. I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework. The Cloud Functions back end is completely agnostic to your front end. If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option.

And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new. From there, we'll CD into the functions directory and then we'll need to set up Node. Js version 8 as the runtime. Open the package JSON, create an object called engines, then set node to a string value of 8. Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config.

This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer. And the last piece of setup is to install Puppeteer with npm. You'll notice that that downloads Chromium, which is over a 100 megabytes in size. And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function. So we're going to beef that up in the next step.

Let's go ahead and open up the index. Ts file and then we'll import Puppeteer in addition to Firebase functions. The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function. But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory. Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function.

The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch. And then there's a number of different configuration options that you can pass into launch. In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly. So that's the equivalent of opening up a browser. The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser.

And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods. The next step is to navigate to an actual URL. In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages. When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits. In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there.

At this point, there's all kinds of different things we can do. We might wanna scrape content from the page. We might want to inject Jquery and make manipulations to it. We can take a screenshot and do pretty much anything else you can do in a regular web browser. But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function.

Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application. So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML. So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter. But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser. Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website.

Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response. So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world. In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that. Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting. So we'll import the serialized function from Rendertron.

Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing. Then we'll write a Cloud Function specifically for rendering pages with Puppeteer. This first render function is going to look almost identical to the one that we just wrote. But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process. And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL.

And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function. The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint. Then what we want to do is determine whether or not the request is coming from a bot or a human. If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page. I have a list of common bots here.

And then we can see the bot on the request by looking at the User Agent header. Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header. So if the length is 0, then we know we're not dealing with a bot. But if it's greater than 0, we know that we should render this page with Puppeteer. If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron.

We can do that by calling the render function that we created previously that handles all the puppeteer stuff. So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page. Then we can take that HTML and send it as the response body from this function. Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back. A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account.

You can either deploy this function or run it locally. Then at this point, I'm assuming that you already have a Firebase Hosting account set up. And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file. We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote. Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app.

And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards. So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface. There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff. I'm gonna go ahead and wrap it up there. If you want to see more advanced puppeteer content, leave me a comment down below.

And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com. You'll get unlimited access to all pro content as well as my full courses and a free copy of my book. Thanks for watching and I'll talk to you soon.
When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment. Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function. If you're new to the channel, like and subscribe. And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course. So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome. Headless Chrome is just a web browser that can run on your server. And Puppeteer is a high level API allowing you to interact with the browser.
It's very intuitive to use and you get direct access to the DOM on the rendered page. So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well. If we go back to episode 66, We used an app called Rendertron to server side render an Angular application. But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container. In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function. I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework. The Cloud Functions back end is completely agnostic to your front end.
animation
If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option. And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new. From there, we'll CD into the functions directory and then we'll need to set up Node. Js version 8 as the runtime. Open the package JSON, create an object called engines, then set node to a string value of 8. Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config. This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer. And the last piece of setup is to install Puppeteer with npm. You'll notice that that downloads Chromium, which is over a 100 megabytes in size. And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function. So we're going to beef that up in the next step. Let's go ahead and open up the index. Ts file and then we'll import Puppeteer in addition to Firebase functions. The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function. But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory. Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function. The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch. And then there's a number of different configuration options that you can pass into launch. In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly. So that's the equivalent of opening up a browser. The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser. And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods. The next step is to navigate to an actual URL. In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages. When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits.
In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there. At this point, there's all kinds of different things we can do. We might wanna scrape content from the page.
internet
networking
We might want to inject Jquery and make manipulations to it. We can take a screenshot and do pretty much anything else you can do in a regular web browser. But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function. Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application.
So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML. So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter.
animation
But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser. Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website. Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response.
So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world. In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that.
graphic design
Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting. So we'll import the serialized function from Rendertron. Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing. Then we'll write a Cloud Function specifically for rendering pages with Puppeteer. This first render function is going to look almost identical to the one that we just wrote. But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process. And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL. And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function. The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint. Then what we want to do is determine whether or not the request is coming from a bot or a human. If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page. I have a list of common bots here. And then we can see the bot on the request by looking at the User Agent header. Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header. So if the length is 0, then we know we're not dealing with a bot. But if it's greater than 0, we know that we should render this page with Puppeteer. If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron.
We can do that by calling the render function that we created previously that handles all the puppeteer stuff. So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page. Then we can take that HTML and send it as the response body from this function.
animation
Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back. A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account. You can either deploy this function or run it locally. Then at this point, I'm assuming that you already have a Firebase Hosting account set up. And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file. We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote. Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app.
And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards. So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface.
social media
google
twitter
There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff. I'm gonna go ahead and wrap it up there. If you want to see more advanced puppeteer content, leave me a comment down below. And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com.
You'll get unlimited access to all pro content as well as my full courses and a free copy of my book. Thanks for watching and I'll talk to you soon.
copyright licenses
When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment. Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function. If you're new to the channel, like and subscribe. And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course. So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome. Headless Chrome is just a web browser that can run on your server. And Puppeteer is a high level API allowing you to interact with the browser. It's very intuitive to use and you get direct access to the DOM on the rendered page. So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well. If we go back to episode 66, We used an app called Rendertron to server side render an Angular application. But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container. In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function. I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework. The Cloud Functions back end is completely agnostic to your front end. If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option. And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new. From there, we'll CD into the functions directory and then we'll need to set up Node. Js version 8 as the runtime. Open the package JSON, create an object called engines, then set node to a string value of 8. Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config. This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer. And the last piece of setup is to install Puppeteer with npm. You'll notice that that downloads Chromium, which is over a 100 megabytes in size. And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function. So we're going to beef that up in the next step. Let's go ahead and open up the index. Ts file and then we'll import Puppeteer in addition to Firebase functions. The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function. But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory. Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function. The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch. And then there's a number of different configuration options that you can pass into launch. In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly. So that's the equivalent of opening up a browser. The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser. And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods. The next step is to navigate to an actual URL. In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages. When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits. In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there. At this point, there's all kinds of different things we can do. We might wanna scrape content from the page. We might want to inject Jquery and make manipulations to it. We can take a screenshot and do pretty much anything else you can do in a regular web browser. But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function. Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application. So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML. So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter. But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser. Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website. Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response. So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world. In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that. Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting. So we'll import the serialized function from Rendertron. Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing. Then we'll write a Cloud Function specifically for rendering pages with Puppeteer. This first render function is going to look almost identical to the one that we just wrote. But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process. And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL. And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function. The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint. Then what we want to do is determine whether or not the request is coming from a bot or a human. If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page. I have a list of common bots here. And then we can see the bot on the request by looking at the User Agent header. Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header. So if the length is 0, then we know we're not dealing with a bot. But if it's greater than 0, we know that we should render this page with Puppeteer. If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron. We can do that by calling the render function that we created previously that handles all the puppeteer stuff. So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page. Then we can take that HTML and send it as the response body from this function. Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back. A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account. You can either deploy this function or run it locally. Then at this point, I'm assuming that you already have a Firebase Hosting account set up. And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file. We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote. Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app. And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards. So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface. There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff. I'm gonna go ahead and wrap it up there. If you want to see more advanced puppeteer content, leave me a comment down below. And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com. You'll get unlimited access to all pro content as well as my full courses and a free copy of my book. Thanks for watching and I'll talk to you soon.
When
when
Cloud
cloud
Functions
functions
2.0
2.0
came
came
out
out
last
last
month,
month
we
we
got
got
access
access
to
to
a
a
whole
whole
bunch
bunch
of
of
cool
cool
new
new
features,
features
such
such
as
as
a
a
Node
node
8
8
runtime,
runtime
custom
custom
memory
memory
and
and
region
region
settings,
settings
and
and
the
the
ability
ability
to
to
use
use
Puppeteer
puppeteer
in
in
the
the
Cloud
cloud
Functions
functions
environment.
environment
Today,
today
we're
we're
going
going
to
to
put
put
all
all
these
these
new
new
features
features
to
to
use
use
so
so
we
we
can
can
server
server
I'd
i'd
render
render
pretty
pretty
much
much
any
any
JavaScript
javascript
application
application
from
from
a
a
Firebase
firebase
function.
function
If
if
you're
you're
new
new
to
to
the
the
channel,
channel
like
like
and
and
subscribe.
subscribe
And
and
if
if
you
you
wanna
wanna
get
get
really
really
good
good
at
at
Cloud
cloud
Functions,
functions
consider
consider
Becoming
becoming
a
a
pro
pro
member
member
to
to
get
get
access
access
to
to
the
the
Cloud
cloud
Functions
functions
master
master
course.
course
So
so
one
one
of
of
the
the
major
major
benefits
benefits
of
of
the
the
new
new
Node
node
8
8
runtime
runtime
in
in
Cloud
cloud
Functions
functions
is
is
that
that
it
it
allows
allows
us
us
to
to
run
run
Puppeteer
puppeteer
and
and
Headless
headless
Chrome.
chrome
Headless
headless
Chrome
chrome
is
is
just
just
a
a
web
web
browser
browser
that
that
can
can
run
run
on
on
your
your
server.
server
And
and
Puppeteer
puppeteer
is
is
a
a
high
high
level
level
API
api
allowing
allowing
you
you
to
to
interact
interact
with
with
the
the
browser.
browser
It's
it's
very
very
intuitive
intuitive
to
to
use
use
and
and
you
you
get
get
direct
direct
access
access
to
to
the
the
DOM
dom
on
on
the
the
rendered
rendered
page.
page
So
so
you
you
can
can
use
use
it
it
for
for
things
things
like
like
web
web
scraping,
scraping
server
server
side
side
rendering,
rendering
end
end
to
to
end
end
testing,
testing
performance
performance
benchmarking,
benchmarking
and
and
probably
probably
a
a
whole
whole
bunch
bunch
of
of
other
other
things
things
as
as
well.
well
If
if
we
we
go
go
back
back
to
to
episode
episode
66,
66
We
we
used
used
an
an
app
app
called
called
Rendertron
rendertron
to
to
server
server
side
side
render
render
an
an
Angular
angular
application.
application
But
but
one
one
of
of
the
the
major
major
drawbacks
drawbacks
of
of
that
that
approach
approach
is
is
that
that
we
we
needed
needed
to
to
build
build
and
and
maintain
maintain
our
our
own
own
Docker
docker
container.
container
In
in
this
this
video,
video
we'll
we'll
build
build
an
an
app
app
that's
that's
equivalent
equivalent
to
to
Rendertron,
rendertron
but
but
on
on
a
a
Firebase
firebase
Cloud
cloud
Function.
function
I'm
i'm
going
going
to
to
be
be
building
building
this
this
function
function
in
in
an
an
Ionic
ionic
four
four
project,
project
but
but
you
you
can
can
build
build
it
it
by
by
self
self
or
or
with
with
any
any
other
other
JavaScript
javascript
framework.
framework
The
the
Cloud
cloud
Functions
functions
back
back
end
end
is
is
completely
completely
agnostic
agnostic
to
to
your
your
front
front
end.
end
If
if
you're
you're
in
in
a
a
brand
brand
new
new
project,
project
run
run
Firebase
firebase
init
init
functions
functions
and
and
then
then
I've
i've
selected
selected
the
the
TypeScript
typescript
option.
option
And
and
it's
it's
very
very
important
important
that
that
you're
you're
up
up
to
to
date
date
on
on
the
the
latest
latest
version
version
of
of
Firebase
firebase
Tools
tools
because
because
all
all
this
this
stuff
stuff
is
is
very
very
new.
new
From
from
there,
there
we'll
we'll
CD
cd
into
into
the
the
functions
functions
directory
directory
and
and
then
then
we'll
we'll
need
need
to
to
set
set
up
up
Node.
node
Js
js
version
version
8
8
as
as
the
the
runtime.
runtime
Open
open
the
the
package
package
JSON,
json
create
create
an
an
object
object
called
called
engines,
engines
then
then
set
set
node
node
to
to
a
a
string
string
value
value
of
of
8.
8
Then
then
another
another
thing
thing
you'll
you'll
wanna
wanna
do
do
if
if
you're
you're
using
using
TypeScript
typescript
is
is
add
add
the
the
DOM
dom
lib
lib
to
to
your
your
TS
ts
config.
config
This
this
is
is
purely
purely
for
for
tooling,
tooling
and
and
it
it
will
will
just
just
give
give
us
us
autocomplete
autocomplete
on
on
anything
anything
related
related
to
to
the
the
DOM
dom
that
that
we're
we're
doing
doing
inside
inside
of
of
Puppeteer.
puppeteer
And
and
the
the
last
last
piece
piece
of
of
setup
setup
is
is
to
to
install
install
Puppeteer
puppeteer
with
with
npm.
npm
You'll
you'll
notice
notice
that
that
that
that
downloads
downloads
Chromium,
chromium
which
which
is
is
over
over
a
a
100
100
megabytes
megabytes
in
in
size.
size
And
and
Chrome,
chrome
in
in
general,
general
requires
requires
a
a
higher
higher
memory
memory
allocation
allocation
than
than
the
the
typical
typical
250
250
megabytes
megabytes
that
that
you
you
would
would
find
find
in
in
a
a
function.
function
So
so
we're
we're
going
going
to
to
beef
beef
that
that
up
up
in
in
the
the
next
next
step.
step
Let's
let's
go
go
ahead
ahead
and
and
open
open
up
up
the
the
index.
index
Ts
ts
file
file
and
and
then
then
we'll
we'll
import
import
Puppeteer
puppeteer
in
in
addition
addition
to
to
Firebase
firebase
functions.
functions
The
the
first
first
thing
thing
I
i
wanna
wanna
show
show
you
you
is
is
how
how
to
to
visit
visit
a
a
page
page
with
with
Puppeteer,
puppeteer
serialize
serialize
the
the
HTML
html
to
to
a
a
string,
string
then
then
send
send
that
that
as
as
the
the
response
response
from
from
the
the
function.
function
But
but
the
the
first
first
thing
thing
we'll
we'll
do
do
is
is
set
set
up
up
our
our
function
function
to
to
run
run
with
with
at
at
least
least
1
1
gigabyte
gigabyte
of
of
memory.
memory
Then
then
after
after
that,
that
we
we
can
can
set
set
up
up
our
our
HTTP
http
function
function
which
which
will
will
be
be
on
on
request
request
giving
giving
us
us
a
a
request
request
and
and
response
response
object,
object
which
which
we
we
can
can
also
also
make
make
an
an
async
async
function.
function
The
the
very
very
first
first
thing
thing
you'll
you'll
do
do
with
with
Puppeteer
puppeteer
is
is
create
create
a
a
Browser
browser
Instance,
instance
which
which
you
you
can
can
do
do
with
with
Puppeteer
puppeteer
Launch.
launch
And
and
then
then
there's
there's
a
a
number
number
of
of
different
different
configuration
configuration
options
options
that
that
you
you
can
can
pass
pass
into
into
launch.
launch
In
in
this
this
case,
case
we'll
we'll
run-in
run-in
Headless
headless
Mode
mode
and
and
I
i
also
also
found
found
that
that
you
you
need
need
the
the
no
no
sandbox
sandbox
flag
flag
for
for
this
this
to
to
work
work
properly.
properly
So
so
that's
that's
the
the
equivalent
equivalent
of
of
opening
opening
up
up
a
a
browser.
browser
The
the
next
next
thing
thing
we'll
we'll
do
do
is
is
create
create
a
a
new
new
page,
page
which
which
can
can
be
be
thought
thought
of
of
as
as
the
the
equivalent
equivalent
of
of
a
a
new
new
tab
tab
in
in
the
the
browser.
browser
And
and
everything
everything
in
in
Puppeteer
puppeteer
is
is
async,
async
so
so
you'll
you'll
notice
notice
that
that
I'm
i'm
awaiting
awaiting
all
all
of
of
these
these
methods.
methods
The
the
next
next
step
step
is
is
to
to
navigate
navigate
to
to
an
an
actual
actual
URL.
url
In
in
this
this
case,
case
I'm
i'm
navigating
navigating
to
to
the
the
Ionic
ionic
app
app
that
that
I
i
built
built
which
which
is
is
just
just
a
a
very
very
simple
simple
set
set
of
of
pages.
pages
When
when
you
you
navigate
navigate
to
to
a
a
page,
page
it's
it's
going
going
to
to
execute
execute
all
all
the
the
JavaScript
javascript
And
and
you
you
can
can
control
control
how
how
long
long
Puppeteer
puppeteer
waits.
waits
In
in
this
this
case,
case
I'm
i'm
setting
setting
it
it
to
to
network
network
idle
idle
0,
0
which
which
just
just
means
means
it'll
it'll
wait
wait
for
for
network
network
request
request
to
to
be
be
idle
idle
for
for
500
500
milliseconds
milliseconds
and
and
then
then
render
render
the
the
page
page
there.
there
At
at
this
this
point,
point
there's
there's
all
all
kinds
kinds
of
of
different
different
things
things
we
we
can
can
do.
do
We
we
might
might
wanna
wanna
scrape
scrape
content
content
from
from
the
the
page.
page
We
we
might
might
want
want
to
to
inject
inject
Jquery
jquery
and
and
make
make
manipulations
manipulations
to
to
it.
it
We
we
can
can
take
take
a
a
screenshot
screenshot
and
and
do
do
pretty
pretty
much
much
anything
anything
else
else
you
you
can
can
do
do
in
in
a
a
regular
regular
web
web
browser.
browser
But
but
if
if
you
you
just
just
need
need
the
the
rendered
rendered
content,
content
which
which
is
is
all
all
we
we
need
need
here,
here
we
we
can
can
say
say
await
await
page
page
dot
dot
content,
content
and
and
then
then
we'll
we'll
send
send
that
that
as
as
the
the
response
response
from
from
this
this
HTTP
http
function.
function
Now
now
we'll
we'll
go
go
ahead
ahead
and
and
serve
serve
this
this
and
and
see
see
if
if
it's
it's
rendering
rendering
the
the
meta
meta
tags
tags
that
that
we
we
have
have
set
set
up
up
on
on
our
our
front
front
end
end
application.
application
So
so
I'm
i'm
using
using
an
an
Ionic
ionic
For
for
App
app
here
here
which
which
has
has
an
an
interesting
interesting
caveat
caveat
with
with
web
web
components,
components
anything
anything
that
that
uses
uses
the
the
Shadow
shadow
DOM
dom
is
is
really
really
hard
hard
to
to
serialize
serialize
back
back
to
to
regular
regular
HTML.
html
So
so
in
in
this
this
demo,
demo
we're
we're
only
only
going
going
to
to
use
use
Puppeteer
puppeteer
to
to
render
render
meta
meta
tags
tags
so
so
we
we
can
can
display
display
our
our
content
content
on
on
social
social
media
media
sites
sites
like
like
Facebook
facebook
and
and
Twitter.
twitter
But
but
that's
that's
really
really
only
only
a
a
concern
concern
if
if
your
your
app
app
uses
uses
the
the
Shadow
shadow
DOM
dom
because,
because
basically,
basically
what
what
we've
we've
built
built
here
here
is
is
a
a
proxy
proxy
browser
browser
that
that
allows
allows
us
us
to
to
visit
visit
a
a
web
web
page
page
without
without
technically
technically
visiting
visiting
that
that
web
web
page
page
on
on
our
our
own
own
browser.
browser
Right
right
now,
now
I'm
i'm
looking
looking
at
at
the
the
rendered
rendered
content
content
for
for
angularfirebase.com,
angularfirebase.com
but
but
I
i
never
never
actually
actually
went
went
to
to
this
this
website.
website
Instead,
instead
I
i
went
went
to
to
the
the
Cloud
cloud
Functions
functions
middleman
middleman
which
which
rendered
rendered
the
the
page
page
with
with
Puppeteer
puppeteer
on
on
the
the
server
server
and
and
then
then
sent
sent
me
me
that
that
serialized
serialized
content
content
as
as
the
the
response.
response
So
so
you
you
basically
basically
now
now
have
have
server
server
side
side
rendering
rendering
for
for
any
any
website
website
on
on
the
the
Internet,
internet
but
but
things
things
aren't
aren't
quite
quite
that
that
simple
simple
in
in
the
the
real
real
world.
world
In
in
the
the
full
full
source
source
code
code
for
for
this
this
project,
project
I've
i've
ported
ported
over
over
the
the
render
render
code
code
from
from
Rendertron
rendertron
which
which
handles
handles
a
a
lot
lot
of
of
the
the
little
little
things
things
that
that
you'll
you'll
need
need
to
to
do
do
when
when
rendering
rendering
pages,
pages
things
things
like
like
setting
setting
the
the
base
base
href
href
for
for
your
your
static
static
resources,
resources
handling
handling
errors,
errors
loading
loading
polyfills
polyfills
for
for
web
web
components,
components
and
and
stuff
stuff
like
like
that.
that
Now
now
the
the
last
last
thing
thing
I
i
wanna
wanna
do
do
is
is
rewrite
rewrite
our
our
functions
functions
So
so
we
we
can
can
set
set
up
up
server
server
side
side
rendering
rendering
with
with
Firebase
firebase
Hosting.
hosting
So
so
we'll
we'll
import
import
the
the
serialized
serialized
function
function
from
from
Rendertron.
rendertron
Then
then
we'll
we'll
set
set
up
up
an
an
appurl,
appurl
which
which
is
is
the
the
location
location
of
of
where
where
your
your
app
app
is
is
deployed
deployed
on
on
Firebase
firebase
Hosting
hosting
or
or
it
it
might
might
be
be
local
local
host
host
5000
5000
for
for
local
local
testing.
testing
Then
then
we'll
we'll
write
write
a
a
Cloud
cloud
Function
function
specifically
specifically
for
for
rendering
rendering
pages
pages
with
with
Puppeteer.
puppeteer
This
this
first
first
render
render
function
function
is
is
going
going
to
to
look
look
almost
almost
identical
identical
to
to
the
the
one
one
that
that
we
we
just
just
wrote.
wrote
But
but
instead
instead
of
of
using
using
page
page
content,
content
we'll
we'll
use
use
our
our
serialized
serialized
method
method
so
so
we
we
have
have
a
a
little
little
more
more
sophisticated
sophisticated
rendering
rendering
process.
process
And
and
for
for
this
this
endpoint,
endpoint
we'll
we'll
allow
allow
the
the
render
render
URL
url
to
to
come
come
through
through
as
as
a
a
query
query
parameter,
parameter
meaning
meaning
it
it
can
can
be
be
passed
passed
directly
directly
through
through
the
the
URL.
url
And
and
the
the
serialized
serialized
method
method
will
will
respond
respond
with
with
the
the
page
page
content
content
as
as
well
well
as
as
a
a
status
status
code,
code
which
which
we
we
can
can
just
just
go
go
ahead
ahead
and
and
send
send
back
back
as
as
the
the
response
response
from
from
this
this
function.
function
The
the
next
next
function
function
we'll
we'll
write
write
is
is
called
called
SSR
ssr
for
for
server
server
side
side
rendering,
rendering
and
and
we're
we're
going
going
to
to
configure
configure
our
our
hosting
hosting
account
account
to
to
reroute
reroute
all
all
requests
requests
to
to
this
this
function
function
endpoint.
endpoint
Then
then
what
what
we
we
want
want
to
to
do
do
is
is
determine
determine
whether
whether
or
or
not
not
the
the
request
request
is
is
coming
coming
from
from
a
a
bot
bot
or
or
a
a
human.
human
If
if
it's
it's
a
a
bot,
bot
we're
we're
going
going
to
to
want
want
to
to
render
render
all
all
of
of
the
the
content
content
on
on
the
the
server
server
so
so
we
we
have
have
the
the
meta
meta
tags
tags
and
and
any
any
other
other
dynamic
dynamic
content
content
loaded
loaded
into
into
the
the
page.
page
I
i
have
have
a
a
list
list
of
of
common
common
bots
bots
here.
here
And
and
then
then
we
we
can
can
see
see
the
the
bot
bot
on
on
the
the
request
request
by
by
looking
looking
at
at
the
the
User
user
Agent
agent
header.
header
Then
then
I'll
i'll
run
run
a
a
filter
filter
on
on
our
our
list
list
of
of
bots
bots
to
to
see
see
if
if
there's
there's
any
any
from
from
this
this
list
list
included
included
in
in
the
the
user
user
agent
agent
header.
header
So
so
if
if
the
the
length
length
is
is
0,
0
then
then
we
we
know
know
we're
we're
not
not
dealing
dealing
with
with
a
a
bot.
bot
But
but
if
if
it's
it's
greater
greater
than
than
0,
0
we
we
know
know
that
that
we
we
should
should
render
render
this
this
page
page
with
with
Puppeteer.
puppeteer
If
if
the
the
user
user
agent
agent
is
is
a
a
bot,
bot
which
which
would
would
happen
happen
if
if
someone
someone
posted
posted
this
this
link
link
in
in
social
social
media,
media
Then
then
we
we
want
want
to
to
go
go
ahead
ahead
and
and
render
render
the
the
page
page
fully
fully
with
with
Rendertron.
rendertron
We
we
can
can
do
do
that
that
by
by
calling
calling
the
the
render
render
function
function
that
that
we
we
created
created
previously
previously
that
that
handles
handles
all
all
the
the
puppeteer
puppeteer
stuff.
stuff
So
so
we'll
we'll
just
just
go
go
ahead
ahead
and
and
await
await
the
the
response
response
from
from
that
that
endpoint,
endpoint
which
which
should
should
be
be
the
the
serialized
serialized
content
content
from
from
the
the
page.
page
Then
then
we
we
can
can
take
take
that
that
HTML
html
and
and
send
send
it
it
as
as
the
the
response
response
body
body
from
from
this
this
function.
function
Otherwise,
otherwise
we
we
want
want
to
to
just
just
fetch
fetch
the
the
root
root
URL
url
from
from
the
the
hosting
hosting
account
account
and
and
send
send
that
that
as
as
the
the
response
response
back.
back
A
a
single
single
page
page
application
application
will
will
get
get
this
this
response
response
and
and
handle
handle
the
the
URL
url
client
client
side,
side
so
so
it'll
it'll
behave
behave
just
just
as
as
if
if
the
the
user
user
visited
visited
this
this
page
page
normally
normally
on
on
a
a
regular
regular
hosting
hosting
account.
account
You
you
can
can
either
either
deploy
deploy
this
this
function
function
or
or
run
run
it
it
locally.
locally
Then
then
at
at
this
this
point,
point
I'm
i'm
assuming
assuming
that
that
you
you
already
already
have
have
a
a
Firebase
firebase
Hosting
hosting
account
account
set
set
up.
up
And
and
if
if
so,
so
you'll
you'll
have
have
it
it
set
set
up
up
as
as
a
a
single
single
page
page
application
application
that
that
looks
looks
something
something
like
like
this
this
where
where
you're
you're
routing
routing
the
the
source
source
to
to
an
an
index
index
HTML
html
file.
file
We're
we're
gonna
gonna
go
go
ahead
ahead
and
and
change
change
that
that
to
to
a
a
function
function
and
and
have
have
it
it
route
route
to
to
the
the
SSR
ssr
function
function
that
that
we
we
just
just
wrote.
wrote
Then
then
the
the
final
final
step
step
is
is
to
to
run
run
Firebase
firebase
deploy,
deploy
which
which
will
will
deploy
deploy
both
both
the
the
functions
functions
and
and
the
the
hosting
hosting
for
for
your
your
client
client
side
side
app.
app
And
and
as
as
a
a
normal
normal
visitor
visitor
to
to
your
your
app,
app
you
you
shouldn't
shouldn't
really
really
notice
notice
any
any
major
major
changes,
changes
but
but
only
only
if
if
you
you
post
post
to
to
a
a
link
link
bot
bot
such
such
as
as
Slack
slack
Twitter
twitter
or
or
whatever,
whatever
it
it
should
should
intercept
intercept
that
that
request
request
and
and
display
display
the
the
meta
meta
tags
tags
necessary
necessary
for
for
rich
rich
social
social
media
media
cards.
cards
So
so
that's
that's
how
how
you
you
set
set
up
up
Puppeteer
puppeteer
in
in
a
a
Cloud
cloud
Function,
function
and
and
we're
we're
really
really
just
just
barely
barely
scratching
scratching
the
the
surface.
surface
There's
there's
all
all
kinds
kinds
of
of
cool
cool
things
things
you
you
can
can
do,
do
one
one
of
of
which
which
is
is
the
the
headless
headless
Chrome
chrome
crawler,
crawler
which
which
is
is
an
an
open
open
source
source
project
project
built
built
on
on
top
top
of
of
Puppeteer
puppeteer
that
that
allows
allows
you
you
to
to
iterate
iterate
over
over
the
the
links
links
in
in
a
a
web
web
page,
page
run
run
jobs
jobs
in
in
parallel,
parallel
export
export
to
to
CSV
csv
and
and
a
a
bunch
bunch
of
of
other
other
advanced
advanced
stuff.
stuff
I'm
i'm
gonna
gonna
go
go
ahead
ahead
and
and
wrap
wrap
it
it
up
up
there.
there
If
if
you
you
want
want
to
to
see
see
more
more
advanced
advanced
puppeteer
puppeteer
content,
content
leave
leave
me
me
a
a
comment
comment
down
down
below.
below
And
and
if
if
you
you
want
want
to
to
take
take
your
your
development
development
to
to
the
the
next
next
level,
level
consider
consider
becoming
becoming
a
a
pro
pro
member
member
at
at
angularfirebase.com.
angularfirebase.com
You'll
you'll
get
get
unlimited
unlimited
access
access
to
to
all
all
pro
pro
content
content
as
as
well
well
as
as
my
my
full
full
courses
courses
and
and
a
a
free
free
copy
copy
of
of
my
my
book.
book
Thanks
thanks
for
for
watching
watching
and
and
I'll
i'll
talk
talk
to
to
you
you
soon.
soon
success
The speaker discusses new CloudGeneration 2.0 features, including access to likerecorded audio and using Puppeteer in the CloudGeneration environment. They also show how to navigate and use the Cloud function for rendering media on social media sites and re-render functions. The speaker explains how to set up server side rendering with Azure Hosting and configure their hosting account to reroute requests. They also discuss how to use the headless Chrome crawler for a better page display and offer a free copy of their book.
Puppeteer on Cloud Functions
When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment. Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function. If you're new to the channel, like and subscribe. And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course. So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome. Headless Chrome is just a web browser that can run on your server. And Puppeteer is a high level API allowing you to interact with the browser. It's very intuitive to use and you get direct access to the DOM on the rendered page. So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well. If we go back to episode 66, We used an app called Rendertron to server side render an Angular application. But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container. In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function. I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework. The Cloud Functions back end is completely agnostic to your front end. If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option. And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new. From there, we'll CD into the functions directory and then we'll need to set up Node. Js version 8 as the runtime. Open the package JSON, create an object called engines, then set node to a string value of 8. Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config. This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer. And the last piece of setup is to install Puppeteer with npm. You'll notice that that downloads Chromium, which is over a 100 megabytes in size. And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function. So we're going to beef that up in the next step. Let's go ahead and open up the index. Ts file and then we'll import Puppeteer in addition to Firebase functions. The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function. But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory. Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function. The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch. And then there's a number of different configuration options that you can pass into launch. In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly. So that's the equivalent of opening up a browser. The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser. And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods. The next step is to navigate to an actual URL. In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages. When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits. In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there. At this point, there's all kinds of different things we can do. We might wanna scrape content from the page. We might want to inject Jquery and make manipulations to it. We can take a screenshot and do pretty much anything else you can do in a regular web browser. But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function. Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application. So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML. So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter. But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser. Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website. Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response. So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world. In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that. Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting. So we'll import the serialized function from Rendertron. Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing. Then we'll write a Cloud Function specifically for rendering pages with Puppeteer. This first render function is going to look almost identical to the one that we just wrote. But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process. And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL. And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function. The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint. Then what we want to do is determine whether or not the request is coming from a bot or a human. If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page. I have a list of common bots here. And then we can see the bot on the request by looking at the User Agent header. Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header. So if the length is 0, then we know we're not dealing with a bot. But if it's greater than 0, we know that we should render this page with Puppeteer. If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron. We can do that by calling the render function that we created previously that handles all the puppeteer stuff. So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page. Then we can take that HTML and send it as the response body from this function. Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back. A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account. You can either deploy this function or run it locally. Then at this point, I'm assuming that you already have a Firebase Hosting account set up. And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file. We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote. Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app. And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards. So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface. There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff. I'm gonna go ahead and wrap it up there. If you want to see more advanced puppeteer content, leave me a comment down below. And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com. You'll get unlimited access to all pro content as well as my full courses and a free copy of my book. Thanks for watching and I'll talk to you soon.
The speaker discusses new CloudGeneration 2.0 features, including access to likerecorded audio and using Puppeteer in the CloudGeneration environment. They also show how to navigate and use the Cloud function for rendering media on social media sites and re-render functions. The speaker explains how to set up server side rendering with Azure Hosting and configure their hosting account to reroute requests. They also discuss how to use the headless Chrome crawler for a better page display and offer a free copy of their book.

        [INST]
        You are youtuber called Fireship you make engaging high-intensity and entertaining coding tutorials and tech news. 
        you covers a wide range of topics relevant to programmers, aiming to help them learn and improve their skills quickly.
        
        Given the title of the video : Puppeteer on Cloud Functions 
        and a small summary : The speaker discusses new CloudGeneration 2.0 features, including access to likerecorded audio and using Puppeteer in the CloudGeneration environment. They also show how to navigate and use the Cloud function for rendering media on social media sites and re-render functions. The speaker explains how to set up server side rendering with Azure Hosting and configure their hosting account to reroute requests. They also discuss how to use the headless Chrome crawler for a better page display and offer a free copy of their book.
        [/INST]
        
        Generate the video : When Cloud Functions 2.0 came out last month, we got access to a whole bunch of cool new features, such as a Node 8 runtime, custom memory and region settings, and the ability to use Puppeteer in the Cloud Functions environment. Today, we're going to put all these new features to use so we can server I'd render pretty much any JavaScript application from a Firebase function. If you're new to the channel, like and subscribe. And if you wanna get really good at Cloud Functions, consider Becoming a pro member to get access to the Cloud Functions master course. So one of the major benefits of the new Node 8 runtime in Cloud Functions is that it allows us to run Puppeteer and Headless Chrome. Headless Chrome is just a web browser that can run on your server. And Puppeteer is a high level API allowing you to interact with the browser. It's very intuitive to use and you get direct access to the DOM on the rendered page. So you can use it for things like web scraping, server side rendering, end to end testing, performance benchmarking, and probably a whole bunch of other things as well. If we go back to episode 66, We used an app called Rendertron to server side render an Angular application. But one of the major drawbacks of that approach is that we needed to build and maintain our own Docker container. In this video, we'll build an app that's equivalent to Rendertron, but on a Firebase Cloud Function. I'm going to be building this function in an Ionic four project, but you can build it by self or with any other JavaScript framework. The Cloud Functions back end is completely agnostic to your front end. If you're in a brand new project, run Firebase init functions and then I've selected the TypeScript option. And it's very important that you're up to date on the latest version of Firebase Tools because all this stuff is very new. From there, we'll CD into the functions directory and then we'll need to set up Node. Js version 8 as the runtime. Open the package JSON, create an object called engines, then set node to a string value of 8. Then another thing you'll wanna do if you're using TypeScript is add the DOM lib to your TS config. This is purely for tooling, and it will just give us autocomplete on anything related to the DOM that we're doing inside of Puppeteer. And the last piece of setup is to install Puppeteer with npm. You'll notice that that downloads Chromium, which is over a 100 megabytes in size. And Chrome, in general, requires a higher memory allocation than the typical 250 megabytes that you would find in a function. So we're going to beef that up in the next step. Let's go ahead and open up the index. Ts file and then we'll import Puppeteer in addition to Firebase functions. The first thing I wanna show you is how to visit a page with Puppeteer, serialize the HTML to a string, then send that as the response from the function. But the first thing we'll do is set up our function to run with at least 1 gigabyte of memory. Then after that, we can set up our HTTP function which will be on request giving us a request and response object, which we can also make an async function. The very first thing you'll do with Puppeteer is create a Browser Instance, which you can do with Puppeteer Launch. And then there's a number of different configuration options that you can pass into launch. In this case, we'll run-in Headless Mode and I also found that you need the no sandbox flag for this to work properly. So that's the equivalent of opening up a browser. The next thing we'll do is create a new page, which can be thought of as the equivalent of a new tab in the browser. And everything in Puppeteer is async, so you'll notice that I'm awaiting all of these methods. The next step is to navigate to an actual URL. In this case, I'm navigating to the Ionic app that I built which is just a very simple set of pages. When you navigate to a page, it's going to execute all the JavaScript And you can control how long Puppeteer waits. In this case, I'm setting it to network idle 0, which just means it'll wait for network request to be idle for 500 milliseconds and then render the page there. At this point, there's all kinds of different things we can do. We might wanna scrape content from the page. We might want to inject Jquery and make manipulations to it. We can take a screenshot and do pretty much anything else you can do in a regular web browser. But if you just need the rendered content, which is all we need here, we can say await page dot content, and then we'll send that as the response from this HTTP function. Now we'll go ahead and serve this and see if it's rendering the meta tags that we have set up on our front end application. So I'm using an Ionic For App here which has an interesting caveat with web components, anything that uses the Shadow DOM is really hard to serialize back to regular HTML. So in this demo, we're only going to use Puppeteer to render meta tags so we can display our content on social media sites like Facebook and Twitter. But that's really only a concern if your app uses the Shadow DOM because, basically, what we've built here is a proxy browser that allows us to visit a web page without technically visiting that web page on our own browser. Right now, I'm looking at the rendered content for angularfirebase.com, but I never actually went to this website. Instead, I went to the Cloud Functions middleman which rendered the page with Puppeteer on the server and then sent me that serialized content as the response. So you basically now have server side rendering for any website on the Internet, but things aren't quite that simple in the real world. In the full source code for this project, I've ported over the render code from Rendertron which handles a lot of the little things that you'll need to do when rendering pages, things like setting the base href for your static resources, handling errors, loading polyfills for web components, and stuff like that. Now the last thing I wanna do is rewrite our functions So we can set up server side rendering with Firebase Hosting. So we'll import the serialized function from Rendertron. Then we'll set up an appurl, which is the location of where your app is deployed on Firebase Hosting or it might be local host 5000 for local testing. Then we'll write a Cloud Function specifically for rendering pages with Puppeteer. This first render function is going to look almost identical to the one that we just wrote. But instead of using page content, we'll use our serialized method so we have a little more sophisticated rendering process. And for this endpoint, we'll allow the render URL to come through as a query parameter, meaning it can be passed directly through the URL. And the serialized method will respond with the page content as well as a status code, which we can just go ahead and send back as the response from this function. The next function we'll write is called SSR for server side rendering, and we're going to configure our hosting account to reroute all requests to this function endpoint. Then what we want to do is determine whether or not the request is coming from a bot or a human. If it's a bot, we're going to want to render all of the content on the server so we have the meta tags and any other dynamic content loaded into the page. I have a list of common bots here. And then we can see the bot on the request by looking at the User Agent header. Then I'll run a filter on our list of bots to see if there's any from this list included in the user agent header. So if the length is 0, then we know we're not dealing with a bot. But if it's greater than 0, we know that we should render this page with Puppeteer. If the user agent is a bot, which would happen if someone posted this link in social media, Then we want to go ahead and render the page fully with Rendertron. We can do that by calling the render function that we created previously that handles all the puppeteer stuff. So we'll just go ahead and await the response from that endpoint, which should be the serialized content from the page. Then we can take that HTML and send it as the response body from this function. Otherwise, we want to just fetch the root URL from the hosting account and send that as the response back. A single page application will get this response and handle the URL client side, so it'll behave just as if the user visited this page normally on a regular hosting account. You can either deploy this function or run it locally. Then at this point, I'm assuming that you already have a Firebase Hosting account set up. And if so, you'll have it set up as a single page application that looks something like this where you're routing the source to an index HTML file. We're gonna go ahead and change that to a function and have it route to the SSR function that we just wrote. Then the final step is to run Firebase deploy, which will deploy both the functions and the hosting for your client side app. And as a normal visitor to your app, you shouldn't really notice any major changes, but only if you post to a link bot such as Slack Twitter or whatever, it should intercept that request and display the meta tags necessary for rich social media cards. So that's how you set up Puppeteer in a Cloud Function, and we're really just barely scratching the surface. There's all kinds of cool things you can do, one of which is the headless Chrome crawler, which is an open source project built on top of Puppeteer that allows you to iterate over the links in a web page, run jobs in parallel, export to CSV and a bunch of other advanced stuff. I'm gonna go ahead and wrap it up there. If you want to see more advanced puppeteer content, leave me a comment down below. And if you want to take your development to the next level, consider becoming a pro member at angularfirebase.com. You'll get unlimited access to all pro content as well as my full courses and a free copy of my book. Thanks for watching and I'll talk to you soon.
        