FAT in Linux

16 Oct 2018

It has to be said that the most popular transfer format (when it comes to file systems) is either FAT32 or NTFS. In today’s article I’ll walk you through creating one of these lowest-common-denominator devices.

First of all, we need to find the device that you want to format. After you’ve attached your pendrive/device, use the lsblk command to determine what your device’s name is.

➜  ~ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    1  29.8G  0 disk

In my case here, it’s called sda.

First of all, we’ll partition the drive using fdisk.

Partitioning

➜  ~ sudo fdisk /dev/sda

Command (m for help): p
Disk /dev/sda: 29.8 GiB, 32015679488 bytes, 62530624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcfaecd67

We’ll create a single partition for the device.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (2048-62530623, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-62530623, default 62530623):

Created a new partition 1 of type 'Linux' and of size 29.8 GiB.

We can take a look at how the partition table now looks with p.

Command (m for help): p
Disk /dev/sda: 29.8 GiB, 32015679488 bytes, 62530624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcfaecd67

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        2048 62530623 62528576 29.8G 83 Linux

We still need to change the type from Linux to W95 FAT32, which has a code of b.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): b
Changed type of partition 'Linux' to 'W95 FAT32'.

We now finish partitioning and move onto formatting. We write the partition table with w.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Formatting

Finally, we use mkfs to create a vfat filesystem on our device’s partition.

➜  ~ sudo mkfs -t vfat /dev/sda1
mkfs.fat 4.1 (2017-01-24)

Remove the USB and then plug it back in. After it mounts automatically, we can verify with df.

Filesystem     Type      Size  Used Avail Use% Mounted on
. . .
. . .
/dev/sda1      vfat       30G   16K   30G   1% /run/media/user/58E6-54A3

Ready to go.

Upgrading AWS Linux to use Java 8

12 Oct 2018

Some applications that you’ll come across will require Java 8 in order to run. By default (as of the time of this article), the Amazon Linux AMI has Java 7 installed.

In order to upgrade these machines so that they are using Java 8, use the following:

# make sure that you install java8 prior to removing java7
sudo yum install -y java-1.8.0-openjdk.x86_64

# update the binary links in-place
sudo /usr/sbin/alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
sudo /usr/sbin/alternatives --set javac /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/javac

# remove java7
sudo yum remove java-1.7

That’s it. You’re now running Java 8.

Trusting a self-signed certificate

12 Oct 2018

When working in development and sandboxes, it can make sense to trust the self-signed certificates that you might be using. This can lower the amount of workflow noise that you might endure.

In today’s article, I’ll take you through generating a certificate; using the certificate (its use-case is terribly simple), and finally trusting the certificate.

Generation

In a previous post titled “Working with OpenSSL”, I took you through a few different utilities available to you within the OpenSSL suite. One of the sections was on generating your own self-signed certificate.

openssl req -x509 -nodes -days 365 -subj '/C=AU/ST=Queensland/L=Brisbane/CN=localhost' -newkey rsa:4096 -keyout server.key -out server.crt

You should receive output which looks like the following:

Generating a RSA private key
.......................................................................................................++++
...............................................................................................................................++++
writing new private key to 'server.key'
-----

On the filesystem now you should have a server.key and server.cer files waiting for you.

Using the certificate

Now we’re going to stand up a web server that uses this key/certificate pair. Using the nginx docker image, we can quickly get this moving with the following nginx.conf.

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  10000;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

  server {
    listen 443;
    index index.html;

    server_name localhost;

    ssl_certificate /opt/server.crt;
    ssl_certificate_key /opt/server.key;

    ssl on;
    root /var/www/public;

    location / {
      try_files $uri $uri/;
    }
  }
}

Starting the server requires the cerificate, key and configuration file to be mounted in. I’ve also exposed 443 here.

docker run --rm \ 
           -ti \
           -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf:ro \
           -v $(pwd)/server.key:/opt/server.key \
           -v $(pwd)/server.crt:/opt/server.crt \
           -p 443:443 \
           nginx

Right now, when we use the curl command without the --insecure switch, we receive the following:

curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Trusting the certificate

We can now use cerutil to work with the NSS database to add this certificate.

If you’re on a brand new system, you may need to create your NSS database. This can be done with the following instructions. Please note, that I’m not using a password to secure the database here.

mkdir -p %HOME/.pki/nssdb
certutil -N -d $HOME/.pki/nssdb --empty-password

With a database created, you can now add the actual certificate itself. You can acquire the certificate with the following script (that uses OpenSSL):

#!/bin/sh
#
# usage:  import-cert.sh remote.host.name [port]
#
REMHOST=$1
REMPORT=${2:-443}
exec 6>&1
exec > $REMHOST
echo | openssl s_client -connect ${REMHOST}:${REMPORT} 2>&1 |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
certutil -d sql:$HOME/.pki/nssdb -A -t "P,," -n "$REMHOST" -i $REMHOST 
exec 1>&6 6>&-

This script is doing a little bit; but most important to see that openssl acquires the certificate for us; then we issue a call to certutil to add the certificate into our store.

Chrome will look for the nss database in $HOME/.pki/nssdb. This is why this folder has been chosen. The -t switch allows you to specify trustargs. Lifted from the manpage:

·   p - Valid peer
·   P - Trusted peer (implies p)
·   c - Valid CA
·   C - Trusted CA (implies c)
·   T - trusted CA for client authentication (ssl server only)

The trust settings are applied as a combination of these characters, in a series of three.

There are three available trust categories for each certificate, expressed in the order SSL, email, object signing for each trust setting.

With the certificate added into the store, we can re-start chrome and hit our website. Chrome no longer complains about the certificate not being trusted.

Debugging node inside of Docker

12 Sep 2018

If your development setup is anything like mine, you’ll like to put all of your applications into their own containers so that they’re isolated from each other. This also gives me a little added guarantee that all of an application’s dependencies are wrapped up nicely before moving the code between environments.

Sometimes, debugging can be a little awkward if this is how you run. In today’s post, I’ll take you through debugging your node apps inside of a container.

Execution

The execution environment is quite simple. We’ll assume that a bash script allows us to start a container which holds our application, and injects any instruction to its console:

docker run --rm -ti \
       -v $(pwd):/usr/src/app \
       -w /usr/src/app \
       -p 3000:3000 \
       -p 9229:9229 \
       node \
       $@

We’ll assume the following:

Our application serves over port 3000
Debugging will run on port 9229
Our application gets mounted to /usr/src/app inside the container

Allowing inspection

Now we need to tell our node process that we want to inspect the process, and allow debugging. This is as simple as using the --inspect switch with your node or in my case nodemon invocations. Here is my debug run script inside of my package.json:

"debug": "node_modules/.bin/nodemon --inspect=0.0.0.0:9229 index.js",

This starts execution, mounting the debug port on 9229 (to align with our docker invocation); it’s also allowing connections from any remote computer to perform debugging. Handy.

Start debugging

Once you’ve issued ./run npm run debug at the console, you’re ready to start debugging.

I use WebStorm for some projects, vim for others; and sometimes will use Chrome Dev Tools with chrome://inspect to be able to see debugging information on screen.

Hope this helps you keep everything isolated; but integrated enough to debug!

Binary dependencies with AWS Lambda

11 Sep 2018

When you’re developing an AWS Lambda, sometimes you’re going to need to install binary package dependencies. Today’s article will take you through the construction of a project that can be deployed into AWS Lambda including your binary dependencies.

Structure

The whole idea here is based on AWS Lambda using Docker to facilite package, deployment, and execution of your function. The standard python:3.6 image available in the standard library is compatible with what we’ll end up deploying.

The structure of your project should have a requirements.txt file holding your dependencies, a standard Dockerfile and of course, your code.

.
├── Dockerfile
├── requirements.txt
└── src
    └── __init__.py

Any depeendencies are listed out by the requirements.txt file.

Docker

We can now bundle our application up, so that it can be used by AWS Lambda.

FROM python:3.6
RUN apt-get update && apt-get install -y zip
WORKDIR /lambda

# add the requirements and perform any installations
ADD requirements.txt /tmp
RUN pip install --quiet -t /lambda -r /tmp/requirements.txt && \
    find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# the application source code is added to the container
ADD src/ /lambda/
RUN find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# pre-compilation into the container
RUN python -m compileall -q /lambda

RUN zip --quiet -9r /lambda.zip .

FROM scratch
COPY --from=0 /lambda.zip /

The docker container is then built with the following:

docker build -t my-lambda .
ID=$(docker create my-lambda /bin/true)
docker cp $ID:/ .

The retrieves the zip file that we built through the process, that’s readily deployable to AWS Lambda.

Older Newer

Cogs and Levers A blog full of technical stuff