Binary dependencies with AWS Lambda

When you’re developing an AWS Lambda, sometimes you’re going to need to install binary package dependencies. Today’s article will take you through the construction of a project that can be deployed into AWS Lambda including your binary dependencies.

Structure

The whole idea here is based on AWS Lambda using Docker to facilite package, deployment, and execution of your function. The standard python:3.6 image available in the standard library is compatible with what we’ll end up deploying.

The structure of your project should have a requirements.txt file holding your dependencies, a standard Dockerfile and of course, your code.

.
├── Dockerfile
├── requirements.txt
└── src
    └── __init__.py

Any depeendencies are listed out by the requirements.txt file.

Docker

We can now bundle our application up, so that it can be used by AWS Lambda.

FROM python:3.6
RUN apt-get update && apt-get install -y zip
WORKDIR /lambda

# add the requirements and perform any installations
ADD requirements.txt /tmp
RUN pip install --quiet -t /lambda -r /tmp/requirements.txt && \
    find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# the application source code is added to the container
ADD src/ /lambda/
RUN find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# pre-compilation into the container
RUN python -m compileall -q /lambda

RUN zip --quiet -9r /lambda.zip .

FROM scratch
COPY --from=0 /lambda.zip /

The docker container is then built with the following:

docker build -t my-lambda .
ID=$(docker create my-lambda /bin/true)
docker cp $ID:/ .

The retrieves the zip file that we built through the process, that’s readily deployable to AWS Lambda.

JSON Web Tokens

The open standard of JSON Web Tokens allows parties to securely transfer claim information. Furthermore, signed tokens allow for verification of the token itself.

One of the most common use-cases of the JWT is for authorization. Once a user’s secrets have been verified, they are issued a JWT containing claim information. This token is then attached to further requests that allows server applications to assert user’s access to services, resources, etc.

JWTs can also be used for adhoc information transfer. A token’s ability to be signed is an important characteristic providing information verifiability between parties.

Structure

A JWT is divided into three primary components:

• Header
• Payload
• Signature

The header contains information about the type of token; this article assumes that JWT is the type and the hashing algorithm. An example header might look something like this:

{
  "alg": "RSA",
  "typ": "JWT"
}

The payload part is expected to have some standard attribute values. These values can fit into the following categories:

A simple payload for a user might look something like this:

{
  "sub": "1",
  "email": "test@test.com",
  "name": "Tester"
}

Finally, the last piece of the JWT is the signature. The signature of the message depends on the hashing algorithm that you’ve selected in your header.

The calculation is going to look something like this:

HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  "secret"
);

For instance, the following JWT:

{
  "alg": "HS256",
  "typ": "JWT"
}

{
  "sub": "1",
  "email": "test@test.com",
  "name": "Tester"
}

Computes down to the following JWT:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwiZW1haWwiOiJ0ZXN0QHRlc3QuY29tIiwibmFtZSI6IlRlc3RlciJ9.mFv3TbmAMWui0w8ofwREb9xFqRRl0_Igahl8tbosHMw

You can see that the token itself is split into three encoded strings: header.payload.signature.

This token is now used in the Authorization header of your HTTP requests!

References

• jwt.io
• rfc-7519

Mounting S3 in Ubuntu

Getting the s3 storage mechanism can be easily integrated into your local linux file system using the s3fs project.

Install the s3fs package as usual:

sudo apt-get install s3fs

Configure authentication using a home-folder credential file called .passwd-s3fs. This file expects data in the format of IDENTITY:CREDENTIAL. You can easily create one of these with the following:

echo MYIDENTITY:MYCREDENTIAL >  ~/.passwd-s3fs
chmod 600  ~/.passwd-s3fs

Finally, mount your S3 bucket into the local file system:

s3fs your-bucket-name /your/local/folder -o passwd_file=/home/michael/.passwd-s3fs

That’t it. You can now use S3 data, just as you would local data on your system.

Create a systemd daemon

Like it or hate it, you’ll find systemd on any modern linux system these days. This management subsystem is the newer replacement for the older init scripts-based systems.

In today’s article, I’m going to show you how to create a daemon that will sit under the management of systemd.

.service file

A service file instructs systemd how to interact with the deployed application. It’s referred to as the service unit configuration. The following is an example:

[Unit]
Description=My Service
After=network.target

[Service]
ExecStart=/usr/bin/my-service
Restart=on-failure

[Install]
WantedBy=multi-user.target

Installation

This file, once you’ve created it gets deployed to systemd with a cp. You also need to notify systemd.

sudo cp my-service.service /lib/systemd/system

sudo systemctl daemon-reload
sudo systemctl enable my-service

Interacting with the service

Once the service is in place, you can start to interact with the daemon using systemctl.

sudo systemctl start my-service
sudo systemctl status my-service

You can also get a hold of any log pushed out to the standard system file descriptors:

journalctl --unit my-service --follow

Removal

Once you no longer need your service, you can remove it simply my rm‘ing the .service file. Of course, you’ll need to disable your service first.

That’s it for creating a systemd service.

vim Tips

Getting the most out of vim is really about committing commonly used key stokes/patterns into muscle memory. Once you have the basics of these movements, your text editing experience becomes a lot more optimised and efficient.

This blog post is just an index of useful keystrokes that I use.

Misc. stuff

Macros

Adding text

Buffer management

Navigation

Split management

NERDTree