Lambda Layers
At re:Invent 2018, AWS announced Lambda Layers, bringing Lambda construct closer to that of an AMI. Put simply, layers are dependencies that can be made available to the function at runtime. Currently there can be 5 layers associated with a function. Layers help with isolating common dependencies, as well as keeping the runtime consistent in terms of versions - just like the AMIs do. Layers are immutable, and are versioned. While this post is focused on Python, the concept of layers applies uniformly regardless of the language (although I feel this is more useful for dynamic languages compared to languages like Java, where the whole artifact is packaged as a part of the deployment artifact). The dependencies are zipped up, uploaded as new layer, or a new version of an existing layer. They can be tied to runtimes (via labels) for validation as well.
In this post I’ll walk through creating a Python layer, and associating it with a function. I’ll use the CLI, aws-shell to accomplish this.
In this walkthrough, we will associate the Python Github Client, PyGithub, with a Layer, so it is available to our lambda function. I needed this for another project which will be blogged later.
$ pwd
/Users/mpandit/work/pygithub
$ python --version
Python 2.7.10
$ pip install pygithub -t .
$ chmod -R 755 .
$ zip -r ../pygithub_layer.zip *
This will create a pygithub_layer.zip in /Users/mpandit/work folder. This archive has the layer that’d be uploaded to AWS.
Next, we will fire up AWS Shell to upload this Layer on AWS. We will add _2_7 in the layer name to identify the version.
$ aws-shell
aws> lambda publish-layer-version --layer-name pygithub_layer_2_7 --zip-file fileb:///Users/mpandit/work/pygithub_layer.zip
{
"Content": {
"CodeSize": 4914196,
"CodeSha256": "VhdxY5459HdfsYBeWdrfCQ0Aedo3+SYaf8h0q6JprX8=",
"Location": "https://prod-04-2014-layers.s3.amazonaws.com/snapshots/:***********:/pygithub_layer_2_7-4f80490e-1d22-4116-b613-abc92b348342?versionId=ET41H_K4V.."
},
"LayerVersionArn": "arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7:1",
"Version": 1,
"Description": "",
"CreatedDate": "2018-12-22T10:02:37.038+0000",
"LayerArn": "arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7"
}
aws> lambda list-layers
{
"Layers": [
{
"LayerName": "pygithub_layer_2_7",
"LayerArn": "arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7",
"LatestMatchingVersion": {
"LayerVersionArn": "arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7:1",
"Version": 1,
"CreatedDate": "2018-12-22T10:02:37.038+0000"
}
}
]
}
aws> lambda list-functions
{
"Functions": [
{
"TracingConfig": {
"Mode": "PassThrough"
},
"Version": "$LATEST",
"CodeSha256": "r3fwRZPDEUt5qgjfBKrKnoTnyZ9a63M6tOw47kvCwi4=",
"FunctionName": "blogpublisher",
"VpcConfig": {
"SubnetIds": [],
"VpcId": "",
"SecurityGroupIds": []
},
"MemorySize": 128,
"RevisionId": "d8476397-5a13-490d-b70c-31ce8ac75612",
"CodeSize": 379,
"FunctionArn": "arn:aws:lambda:us-east-1:***********:function:blogpublisher",
"Handler": "lambda_function.lambda_handler",
"Role": "arn:aws:iam:***********:role/lambda_basic_execution",
"Timeout": 100,
"LastModified": "2018-12-22T09:20:25.436+0000",
"Runtime": "python2.7",
"Description": ""
}
]
}
aws> lambda update-function-configuration --layers arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7:1 --function-name blogpublisher
{
"Layers": [
{
"CodeSize": 4914196,
"Arn": "arn:aws:lambda:us-east-1:***********:layer:pygithub_layer_2_7:1"
}
],
"FunctionName": "blogpublisher",
"LastModified": "2018-12-22T10:09:41.946+0000",
"RevisionId": "ecc6d869-c73f-4270-8042-f8603f050a4c",
"MemorySize": 128,
"Version": "$LATEST",
"Role": "arn:aws:iam:***********:role/lambda_basic_execution",
"Timeout": 100,
"Runtime": "python2.7",
"TracingConfig": {
"Mode": "PassThrough"
},
"CodeSha256": "r3fwRZPDEUt5qgjfBKrKnoTnyZ9a63M6tOw47kvCwi4=",
"Description": "",
"VpcConfig": {
"SubnetIds": [],
"VpcId": "",
"SecurityGroupIds": []
},
"CodeSize": 379,
"FunctionArn": "arn:aws:lambda:us-east-1:***********:function:blogpublisher",
"Handler": "lambda_function.lambda_handler"
}
The above commands confirm that the layer is uploaded, and associated with the function blogpublisher. Now we can code the function and use the layer, which is made available to us at runtime.
I had to add
/opt/to the path, as the runtime expands the layers in/opt. I could not get it to work without adding this, probably coz I am a Python n00b. There has to be a better way, but this works too.
You’ll need a Github personal access token with proper permissions, which can be created here.
import sys
sys.path.append("/opt/") # This line took me 2 hours to figure out.
from github import Github # This is coming from the layer
def lambda_handler(event, context):
g = Github('replace_with_your_personal_access_token')
for repo in g.get_user().get_repos():
print(repo.name)
Once ran, we can see the repositories, like so -
START RequestId: b0c02335-0892-11e9-a1cb-6fdc63bf3cc9 Version: $LATEST
dockerfiles
dockerfiles-mirror
dotfiles
elasticmq-docker
helloworld-api
lambda-proxy
lobster1234.github.io
localstack
m101j-hw3-23
munin-mongo-collections
my-aws-infrastructure
og-aws
play-salat
RxJava
Scala-For-the-Impatient-Exercises
scalatra-website
SimianArmy
sparkjava-archetypes
SVCC2013
swagger-core
vagrant-demo
END RequestId: b0c02335-0892-11e9-a1cb-6fdc63bf3cc9
REPORT RequestId: b0c02335-0892-11e9-a1cb-6fdc63bf3cc9 Duration: 1559.17 ms Billed Duration: 1600 ms Memory Size: 128 MB Max Memory Used: 44 MB