AWS Lambda in 5 easy steps!
Photo by Marius Masalar on Unsplash

I’m a very curious person and I’ve always been attracted by the cloud services offered by Amazon Web Services (AWS), and the truth is that storage in S3 and Lambdas are tools I learned at full speed developing my final project at Iron Hack.

A few days ago, at work, my partner David taught me how to launch a AWS Lambda to extract some data from the network with a script written in Python. And I think it’s time to show you everything I’ve learned and discovered along the way.

Today we will create an AWS Lambda to run any script written in Python every 24h. But if you want to run it every minute, you can do that too! ;)

Step 1: Python script to get data

First, we will prepare a script in Python (Datatauscraper.py) to obtain the title and the website of the first element of https://datatau.net/ using the Requests and Beautiful Soup libraries. This is the script to do the scraping of the data from Datatau site.

from datetime import datetime import requests from bs4 import BeautifulSoup



def scrapeDatatau():

try:

url = 'https://datatau.net/'

page = requests.get(url)

soup = BeautifulSoup(page.content, 'html.parser')

title = soup.find_all("td", {"class": "title"})

title_val = [tit.get_text() for tit in title][1]

title_val = re.sub('\

','', title_val)

TimeNow = datetime.now()



return {

'date': str(TimeNow),

'title': title_val,

}

except Exception as e: print(e)

In our case, we will get the title and the website of the first article that appears on the website of Datatau. Today, what we get is the following result:

{'date': '2020-07-19 19:34:41.320461',

'title': 'Product Data Science Interview Guide (interviewquery.com)'}

Step 2: Create the Python zip file for uploading to AWS

To host our Python script in the AWS Lambda service, we need to prepare the Python modules with the script in the zip file.

First, we have to install all modules locally with “pip and -t .” tag to specify the one saved in a local directory. In this example, we need the requests and Beautifulsoup4 modules. You can run the following command to install them locally:

$ pip3 install beautifulsoup4 requests -t .

Then use the zip command to create a zip file from this folder like this:

$ zip -r python_package.zip *

Step 3: Prepare the S3 Bucket

As we will save our result in a S3 Bucket, we will create one in the AWS console through the web: https://s3.console.aws.amazon.com/.

Click on “Create bucket” and assign a meaningful name for the project. You can select any region according to your preferences. In this case, I will choose Paris. And I give the name datatau-extract to save the data contained in DataTau.

When you have done everything, just click on the “Next” button to continue. The following steps can be left by default and go to next until you find yourself in step 4 with the summary and the “Create Bucket” button, press the button for the last time to create your own bucket.

Step 4: Create a Role for AWS Lambda

We need to generate a Role so that our Lambda AWS can access the S3 Bucket that we just created in the previous step (datatau-extract).

We do it from the AWS IAM console (https://console.aws.amazon.com/iam/home)

Select “ROLES” in the left menu (Dashboard/Access Management) and click on “Create role” and then select “Lambda” in the case of use. Find a policy called “AWSLambdaExecute” that will allow Lambda to access the S3 Bucket. After confirmation, you can name this Role as “lambda-s3-role”.

Step 5: Upload our Python script to the AWS Lambda layers

Now, everything is almost ready. It’s time to upload our Python package in *.zip format to the Lambda layer to make our Python script executable in Lambda. We can do this by navigating to the AWS Lambda page (https://console.aws.amazon.com/lambda/) and clicking on the “Layers” menu and then clicking on “Create Layer”, giving it a name (I have put Datatau-scraping-test), selecting our *.zip file, and the Python version in which we want to run our script.

Final step: Create your brand new Lambda function

Now, let’s create a Lambda function to execute the script we have prepared. Click here to enter the AWS Lambda console and click on the menu “Functions” and now click on “Create function”. Then select “Create from scratch”, write a name for your function (I have called it DatatauData), choose “Python 3.x runtime”. It’s important to select the existing role “lambda-s3-role” that you created in Step 4 to allow this function to access the S3 cube. They are linked so it’s important that you don’t forget this step. To finish, just press the “Create function” button.

Female software engineer codes at computer by ThisisEngineering RAEng

Now we just have to click on “Layers”, then we mark the radio button that says “select from a list of compatible layers” and in the drop down menu we choose the “Datatau-scraping-test” layer that we have created in Step 5. This action will allow our function to be able to read and call the Python script we created in Step 1. We choose the version of it (in my case number 3), and click on the button in the lower right corner where it says: “Add layer”.

Go to the code part of the Lambda Function that you will see below. We will create a Python script that will allow the Lambda to import the function “scrapeDatatau” from the Python script “Datatauscraper.py” of the “Datatau-scraping-test” layer.

Now, we are going to import the “boto3 library” to access the S3 service.

This is the Python script that will be executed every time you wake up to your Lambda function to scrape the Datatau data and save it in the S3 cube.

If you look at the example I wrote below, you should just copy it and remember to rename the S3 cube to the same name you created in Step 3.

import json

import boto3

from Datatauscraper import scrapeDatatau def lambda_handler(event, context):

s3 = boto3.resource('s3')

print ("Extracting the data from Datatau...")

datatau_data = scrapeDatatau()

BUCKETS3_NAME = "datatau-extract"

DATE = f"{datatau_data['date']}"

OUTPUT_NAME = f"dataKeyTest{DATE}.json"

OUTPUT_BODY = json.dumps(datatau_data)

print (f"Saving your info to S3 {BUCKETS3_NAME}...")

s3.Bucket(BUCKETS3_NAME).put_object(Key = OUTPUT_NAME, Body =\ OUTPUT_BODY)

print (f"Information successfully extracted at {DATE}")

The way we define lambda is VERY IMPORTANT

It is important for your lambda to work that you make sure that you define your lambda with “def lambda_handler(event, context):” otherwise nothing will work in AWS when you upload the code.

You can test your new Lambda by clicking the “Test” button. Don’t worry about anything because it works, and it does it really well, the AWS servers work for you and they do it great.

Now that I remember, I’m going to give you a little tip from my first class on AWS Lambdas for a web scraping project (thanks David!).

“A great advice not to go crazy at the beginning when you do your first tests: measure well the runtimes, 10 minutes maximum. If you see that your script will have a runtime higher than 10 minutes, you’d better consider segmenting your scripts and run it with several Lambdas”. — David Ortega (CTO at Roams.es)

Before I forget, it is important that you check the RAM too, I have set it to 3000 MB and a 5 second timeout but each case will be different and you will have to experiment with your script. But, in my case, it doesn’t need that much either, it’s just an example.

OH PARTY!

Oh yes, we’ve reached the last step, where things happen, where magic becomes amazing!

Now we are going to program our Lambda so that it can be executed automatically every day (in my case), but you can program it to run every hour, or even every minute; you are free to choose the times you prefer. It is not the same to extract data from a sports newspaper, from the weather data for each day or the minute-by-minute data from the US presidential elections.

Do Something Great by Clark Tibbs

From my view, before starting with this task and being the last one, it is important that you familiarize yourself with the Cron expressions. It depends on them that your task is executed exactly in the moment that you want. You have the control of your tasks if you know how to program a CRON expression.

Let’s see the syntax of a CRON expression:

A CRON expression contains five (or six) elements separated by a space:

minutes

hours

day of the month

month

weekday

year (optional)

You can use the character/code * to indicate that this work should work at all times. You can also set multiple values using commas and set ranges using special dashes and combiners:

* * * * * // Run every minute 0 * * * * // Run on the hour, every hour 0 23 * * * // Run at 11 p.m. every day 0 0,12 * * 6 // Run at midnight and midday every Saturday 30 2 */2 * * // Run every second day at 2.30 a.m.

Special characters:

* (Any) -The work is executed in each time unit.

(Values) - Multiple value of the time unit.

- (Range) - Range of the time unit.

/ (Increment) - Specify incremental values of the time unit.

? (No specific value) - Useful when you need to specify something in one of the two fields where the character is allowed, but not in the other.

# (occurrence) - Used to specify the “N-th” occurrence of a day of the week in the month, e.g. “First Monday of the month” can be indicated as “1#1”

L (the last) - If applied in the ‘day-of-month’ field, it means the last day of the month, i.e. ‘31st January. In the ‘weekday’ field, specify the ‘last day of the week’ as ‘5L’, the ‘last Friday’.

I’m sure you’ll be left with some doubt, so one of the first stops in your reference will be to consult “Schedule Expressions for Rules” from the CloudWatch documentation on Amazon here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html

Photo by Malvestida Magazine on Unsplash

If you have arrived safely here, that’s wonderful because I’m going to smooth your way a bit.

In the last step, we’re going to generate an expression that extracts the information generated by our scraping script every day at 0:00 am. But we are going to generate it thanks to a little trick ;) shhhhhhhhhh!!! it’s here — ->> https://crontab.guru/ with this tool we will create our exact Crontab.

Well, now that we’ve warmed up a bit, I’ll just tell you that this task is like a Crontab and we’re going to do it thanks to CloudWatch from AWS from here https://console.aws.amazon.com/cloudwatch

First, you have to click on “Event/ Rules” and on the button “Create a rule”, choose “Scheduled” and “Cron expression”. I have chosen to do it “Every day from Monday to Friday at 0:00 am” and the resulting expression is 0 0 * * 1–5 (https://crontab.guru/#0_0_*_*_1-5).

Then, in the left area, it says “Targets”, there you have to click on the “Add target button” and select our Lambda function (DatatauData) that we created in Step number 6. We leave the rest of the values “by default” and click on the “Configure Details” button.

Et voilà! You’ll have your Lambda scheduled to run every day from Monday to Friday at 0:00 am.

Conclusion

It is important that you become well acquainted with your surroundings, that you repeat the above actions from beginning to end, learn from them, and that you create as many Lambdas as you need. Only then you will learn the true power of the AWS Lambdas, learn about another Amazon Cloud Solutions and have fun with the new professional and personal challenges.

I hope you enjoy this article as much as I have enjoy writing it. If you liked my article, learned something new or found it useful, I encourage you to give me a “Clap for it”. If you want to read my newest articles, you can also follow me here.

Thanks a lot!
['Borja Uría']
['AWS Lambda', 'Cloud Computing', 'Python', 'AWS', 'Web Development']