Running Serverless Code on AWS Lambda

This post is a follow-up to one of my recent blog posts.

There’s lots of guides out there on this topic – actually too many because lots of them don’t work 100%. Trust me, I went through weeks of hell figuring it out. This is the simplest, most straightforward guide I’ve found. It only required a few tweaks for me to make it work.

Here are the steps I took:

In AWS Lambda, you have to use Docker to create a deployment package. The guide I follow was based on Python 3.6, but it still worked when I ran it in the Python 3.7 runtime environment.

As mentioned before, this method requires Docker, so you’ll need to install it if you have never used it before. And don’t worry if you’ve never used Docker before. It’s pretty easy to get started on and you don’t need to know that much to get a lot accomplished.

Once you have Docker installed, fire up a terminal and type in the following command, which is different the one mentioned in the guide I used and comes from here:

docker run -it dacut/amazon-linux-bi

This will first look for the image locally on your computer and naturally, when it does not find it, it will look for it on Docker Hub, pull it, and then start the container in your shell. The image we are pulling is an Amazon Linux instance with Python 3.6 installed on it, which is exactly what we need to build our deployment package since AWS Lambda functions using Python 3.6 are executed in an Amazon Linux environment.

When the download is finished, you will notice your prompt change and you will be in the root directory of the Amazon Linux environment. If you do ls, you will notice that you are not in your local environment. And if you are on a Mac like me, you can run uname to verify that you are running a Linux environment:

Now, create a new directory (mkdir) and cd into it. I’ll call my directory (aka folder) “lambda” for this example. Then run the pip command inside of the newly created directory. See all three steps below:

mkdir lambda
cd lambda
pip3 install <PACKAGE_NAME> -t ./

FYI, the variable <PACKAGE_NAME> should be replaced with any package, such as tweepy or numpy or pandas or any other code package(s) that your code depends on.

This will install your package in the current directory. If you have multiple packages, you can just separate the package names with spaces and run it all on the same line.

pip3 install tweepy numpy pandas -t ./

Once all your packages are installed, in the same directory, run the following:

# (you can name the .zip file whatever you prefer. in this case, it's named lambda.zip)

zip -r lambda.zip * 

This zips up all your packages into a zip file. The next step is to simply copy this zip file from your Docker container onto your local machine.

The code below helps you do that by returning the Docker container ID. The first part lists the containers, then the grep command filters on those with the name “dacut” in it (which is unique to the image we pulled), and finally the awk command returns only the first column, which is the container ID we want. Given all this, we can finally run the following command to copy the zip file to our local machine. In another Terminal window (from your local machine), run:

docker cp $(docker ps -a | grep "dacut" | awk '{print $1}'):/lambda/lambda.zip <PATH_TO_YOUR_LOCAL_DIRECTORY_OF_CHOICE>

In other words, the format is:

docker cp <CONTAINER_ID>:<DOCKER_PATH_TO_ZIP_FILE> <LOCAL_PATH>

In my case, the local path (referred to in the two code examples above as <PATH_TO_YOUR_LOCAL_DIRECTORY_OF_CHOICE> and <LOCAL_PATH>) looks like this, since I’m on a Mac:

/Users/theNameOfMyUserProfile/Coding/AWS-Lambda/projectName
Here's the full example:
docker cp $(docker ps -a | grep "dacut" | awk '{print $1}'):/lambda/lambda.zip /Users/myUserProfile/Coding/AWS-Lambda/projectName

Note: If the above code doesn’t work for you, you may have multiple images. You’ll have to replace $(docker ps -a | grep “dacut” | awk ‘{print $1}’) with the correct container ID. You can figure out the current or most recent ID with these commands:

docker ps -a
docker ps -a | grep “dacut” | awk ‘{print $1}’

See the 2nd link from the top of this post for more info. If you were to write out all of the code for this, your solution might look something this (to get the first field from the first line of results from the list of containers):

docker cp $(docker ps -a | grep "dacut" | awk 'FNR == 1 {print $1}'):/lambda/lambda.zip /Users/myUserProfile/Coding/AWS-Lambda/myProjectName

In your case, the local path could look something like this, depending on where you want to save this project on your computer:

/Users/johnDoe/Documents

Now you have the zip file in a directory on your local machine. cd to that local directory and add your lambda_function.py to it by just running:

zip -ur lambda.zip <PATH_TO_LAMBDA_FUNCTION_FILE>

In this case, <PATH_TO_LAMBDA_FUNCTION_FILE> is what’s typically referred to as the lambda_function.py file in the AWS documentation. In my experience, you want this file to be located in the same folder as your lambda.zip file (in this blog it’s referred to as the lambda folder). In that specific case, the way to write the code above would be:

zip -ur lambda.zip lambda_function.py

And done! You now have a Python 3.6 3.7 deployment package that is ready to be deployed to AWS! (Make sure that your runtime is set to Python 3.7 in the AWS Lambda dashboard. I got a complex looking numpy error when I tried to use Python 3.6).

Upload the updated lambda.zip file to an AWS S3 bucket, copy the S3 file path into the AWS Lambda dashboard. Type in lambda_function.lambda_handler for the Handler setting (if that matches with your file name and handler name). Set a trigger to start your function (or place it on a scheduled routine).

Give your Lambda code a basic memory and timeout setting. For the Twitter bot I made using tweepy, it works on 256mb and 1 second. You may need more for your code.

From the AWS Lambda dashboard, click Save. Then click Test (at least that works for me to get the code started).

Happy coding!

*Note: If you ever want to edit your code after the fact, simply repeat the section in blue above.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s