Cogs and Levers A blog full of technical stuff

Shell Tricks

Sometimes you can be just as productive using your shell as you are in any programming environment, you just need to know a couple of tricks. In this article, I’ll walk through some basic tips that I’ve come across.

Reading input

You can make your scripts immediately interactive by using the read instruction.

#/bin/bash

echo -n "What is your name? "
read NAME

echo "Hi there ${NAME}!"

String length

You can get the length of any string that you’ve stored in a variable by prefixing it with #.

#/bin/bash

echo -n "What is your name? "
read NAME

echo "Your name has ${#NAME} characters in it"

Quick arithmetic

You can perform some basic arithmetic within your scripts as well. The value emitted with the # character is an integral value that we can perform tests against.

#/bin/bash

echo -n "What is your name? "
read NAME

if (( ${#NAME} > 10 )) 
then
  echo "You have a very long name, ${NAME}"
fi

Substrings

String enumeration will also allow you to take a substring directly. The format takes the form of ${VAR:offset:length}.

Passing positive integers for offset and length will make substring operate from the leftmost side of the string. Negative numbers provide a reverse index, from the right.

STR="Scripting for the win"

echo ${STR:10:3}
# for

echo ${STR: -3}
# win

echo ${STR: -7: 3}
# the

Replacement

It’s common place to be able to use regular expressions to make substitutions where needed, and they’re available to you at the shell as well.

#!/bin/bash

STR="Scripting for the win"

echo ${STR/win/WIN}
# Scripting for the WIN

Finishing up

There’s lots more that you can do just from the shell, without needing to reach for other tools. This is only a few tips and tricks.

Mutual TLS (mTLS)

Introduction

TLS has forever played a very large part in securing internet communications. Secure Socket Layer (SSL) filled this space prior to TLS coming to the fore.

In today’s article, I’m going to walk through an exercise of mTLS which is just an extension of TLS.

CA

First of all, we need a certificate authority (CA) that both the client and the server will trust. We generate these using openssl.

openssl req -new -x509 -nodes -days 365 -subj '/CN=my-ca' -keyout ca.key -out ca.crt

This now puts a private key in ca.key and a certificate in ca.crt on our filesystem. We can inspect these a little further with the following.

openssl x509 --in ca.crt -text --noout

Looking at the output, we see some interesting things about our CA certificate. Most importantly the X509v3 Basic Constraints value is set CA:TRUE, telling us that this certificate can be used to sign other certificates (like CA certificates can).

Server

The server now needs a key and certificate. Key generation is simple, as usual:

openssl genrsa -out server.key 2048

We need to create a certificate that has been signed by our CA. This means we need to generate a certificate signing request, which is then used to produce the signed certificate.

openssl req -new -key server.key -subj '/CN=localhost' -out server.csr

This gives us a signing request for the domain of localhost as mentioned in the -subj parameter. This signing request now gets used by the CA to generate the certificate.

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out server.crt

Inspecting the server certificate, you can see that it’s quite a bit simpler than the CA certificate. We’re only able to use this certificate for the subject that we nominated; localhost.

Client

The generation of the client certificates is very much the same as the server.

# create a key
openssl genrsa -out client.key 2048

# generate a signing certificate
openssl req -new -key client.key -subj '/CN=my-client' -out client.csr

# create a certificate signed by the CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out client.crt

The subject in this case is my-client.

The -CAcreateserial number also ensures that we have unique serial numbers between the server and client certificates. Again, this can be verified when you inspect the certificate.

# Server serial number
        Serial Number:
            5c:2c:47:44:2c:13:3b:c9:56:56:99:37:3f:c9:1e:62:c4:c7:df:20

# Client serial number
        Serial Number:
            5c:2c:47:44:2c:13:3b:c9:56:56:99:37:3f:c9:1e:62:c4:c7:df:21

Only the last segment was incremented here. You get the idea though. Unique.

Appliation

Now, we setup a basic node.js server that requires mTLS.

const https = require('https');
const fs = require('fs');

const hostname = 'localhost';
const port = 3000;

const options = { 
    ca: fs.readFileSync('ca.crt'), 
    cert: fs.readFileSync('server.crt'), 
    key: fs.readFileSync('server.key'), 
    rejectUnauthorized: true,
    requestCert: true, 
}; 

const server = https.createServer(options, (req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Most important here is that the server’s options specify rejectUnauthorized as well as requestCert. This will force the mTLS feedback look back to the client.

A curl request now verifies that the solution is secured by this system of certificates.

curl --cacert ca.crt --key client.key --cert client.crt https://localhost:3000

The client’s key, certificate, and the ca cert accompany a successful request. A request in any other format simply fails as the authentication requirements have not been met.

Find the listening process for a port

Introduction

In networking, a port is assigned as a logical entity that a socket is established on. These sockets are owned by processes in your operation system. From time to time, it can be unclear which process owns which socket (or who is hogging which port).

In today’s article, I’ll take you through a few techniques on finding out who is hanging onto particular ports.

netstat

netstat is a general purpose network utility that will tell you about activity within your network interfaces.

netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

If you can not find netstat installed on your system, you can normally get it from the net-tools package.

The following command will give you a breakdown of processes listening on port 8080, as an example:

➜  netstat -ltnp | grep -w ':8080'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::8080                 :::*                    LISTEN      -                   

An important message appears here. “Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.”. There will be processes invisible to you unless you run this command as root.

Breaking down the netstat invocation:

  • l will only show listening sockets
  • t will only show tcp connections
  • n will show numerical addresses
  • p will show you the PID

You can see above, that no process is shown. Re-running this command as root:

➜  sudo netstat -ltnp | grep -w ':8080'
tcp6       0      0 :::8080                 :::*                    LISTEN      2765/docker-proxy

lsof

lsof will give you a list of open files on the system. Remember, sockets are just files. By using -i we can filter the list down to those that match on an internet address.

➜  sudo lsof -i :8080
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 2765 root    4u  IPv6  36404      0t0  TCP *:http-alt (LISTEN)

fuser

fuser is a program that has overlapping responsibilities with the likes of lsof.

fuser — list process IDs of all processes that have one or more files open

You can filter the list down directly with the command:

➜  sudo fuser 8080/tcp
8080/tcp:             2765

This gives us a PID to work with. Again, note this is run as root. Now all we need to do is to tranform this PID into a process name. We can use ps to finish the job.

➜  ps -p 2765 -o comm=
docker-proxy

PostgreSQL Data Access with Haskell

Introduction

PostgreSQL is a very popular relational database which has quite a few different data access libraries available for the Haskell programming language.

Today’s article aims to get you up and running, executing queries against PostgreSQL from your Haskell environment with the least amount of hassle.

postgresql-simple

The first library that we’ll go through is postgresql-simple. This library has a very basic interface, and is really simple to get up an running.

A mid-level client library for the PostgreSQL database, aimed at ease of use and high performance.

Prerequisites

Before you get started though, you’ll need libpq installed.

pacman -S postgresql-libs

Now you’re ready to develop.

You’ll need to add a dependency on the postgresql-simple library to your application. The following code will then allow you to connect to your PostgreSQL database, and ru a simple command.

Hello, Postgres!

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Database.PostgreSQL.Simple

localPG :: ConnectInfo
localPG = defaultConnectInfo
        { connectHost = "172.17.0.1"
        , connectDatabase = "clients"
        , connectUser = "app_user"
        , connectPassword = "app_password"
        }

main :: IO ()
main = do
  conn <- connect localPG
  mapM_ print =<< (query_ conn "SELECT 1 + 1" :: IO [Only Int])

When your application successfully builds and executes, you should be met with the following output:

Only {fromOnly = 2}

Walking through this code quickly, we first enable OverloadedStrings so that we can specify our Query values as literal strings.

localPG :: ConnectInfo
localPG = defaultConnectInfo
        { connectHost = "172.17.0.1"
        , connectDatabase = "clients"
        , connectUser = "app_user"
        , connectPassword = "app_password"
        }

In order to connect to Postgres, we use a ConnectInfo value which is filled out for us via defaultConnectInfo. We just override those values for our examples. I’m running PostgreSQL in a docker container, therefore I’ve got my docker network address.

conn <- connect localPG

The localPG value is now used to connect to the Postgres database. The conn value will be referred to after successful connection to send instructions to.

mapM_ print =<< (query_ conn "SELECT 1 + 1" :: IO [Only Int])

Finally, we run our query SELECT 1 + 1 using the query_ function. conn is passed to refer to the connecion to execute this query on.

With this basic code, we can start to build on some examples.

Retrieve a specific record

In the Hello, World example above, we were adding two static values to return another value. As exampeles get more complex, we need to give the library more information about the data that we’re working with. Int is very well known already, and already has mechanisms to deal with it (along with other basic data types).

In the client database table we have a list of names and ids. We can create a function to retrieve the name of a client, given an id:

retrieveClient :: Connection -> Int -> IO [Only String]
retrieveClient conn cid = query conn "SELECT name FROM client WHERE id = ?" $ (Only cid)

The Query template passed in makes use of the ? character to specify where substitutions will be put. Note the use of query rather than query_. In this case, query also accepts a Tuple containing all of the values for substitution.

Using the FromRow type class, our code can define a much stronger API. We can actually retrieve client rows from the database and convert them into Client values.

We need FromRow first:

import Database.PostgreSQL.Simple.FromRow

The Client data type needs definition now. It’s how we’ll refer to a client within our Haskell program:

data Client = Client { id :: Int, name :: String }
  deriving (Show)

The Client data type now gets a FromRow instance, which allows postgresql-simple to use it.

instance FromRow Client where
  fromRow = Client <$> field <*> field

In order of the fields definitions, we give fromRow definition. The retrieveClient function only changes to broaden its query, and change its return type!

retrieveClient :: Connection -> Int -> IO [Client]
retrieveClient conn cid = query conn "SELECT id, name FROM client WHERE id = ?" $ (Only cid)

Create a new record

When creating data, you can use the function execute. The execute function is all about execution of the query without any return value.

execute conn "INSERT INTO client (name) VALUES (?)" (Only "Sam")

Extending our API, we can make a createClient function; but with a twist. We’ll also return the generated identifier (because of the id field).

createClient :: Connection -> String -> IO [Only Int64]
createClient conn name =
  query conn "INSERT INTO client (name) VALUES (?) RETURNING id" $ (Only name)

We need a definition for Int64. This is what the underlying SERIAL in PostgreSQL will translate to inside of your Haskell application.

import Data.Int

We can now use createClient to setup an interface of sorts fo users to enter information.

main :: IO ()
main = do
  conn <- connect localPG
  putStrLn "Name of your client? "
  clientName <- getLine
  cid <- createClient conn clientName
  putStrLn $ "New Client: " ++ (show cid)

We’ve created a data creation interface now.

Name of your client?
Ringo
New Client: [Only {fromOnly = 4}]

Update an existing record

When it comes to updating data, we don’t expect much back in return aside from the number of records affected by the instruction. The execute function does exactly this. By measuring the return, we can convert the row count into a success/fail style message. I’ve simply encoded this as a boolean here.

updateClient :: Connection -> Int -> String -> IO Bool
updateClient conn cid name = do
  n <- execute conn "UPDATE client SET name = ? WHERE id = ?" (name, cid)
  return $ n > 0

Destroying records

Finally, destroying information out of the database will look a lot like the update.

deleteClient :: Connection -> Int -> IO Bool
deleteClient conn cid = do
  n <- execute conn "DELETE FROM client WHERE id = ?" $ (Only cid)
  return $ n > 0

execute providing the affected count allows us to perform the post-execution validation again.

Summary

There’s some basic operations to get up and running using postgresql-simple. Really looks like you can prototype software all the way through to writing fully blown applications with it.

Really simple to use.

Add info at build time to go binaries

Introduction

Sometimes it can be useful to capture information about your environment at build time, and have this information injected into the binary that you’re building. Some examples centre around versioning, where it might make sense to capture git commit hashes or build serials.

An example program

package main

import "fmt"

var gitCommit string
var buildSerial string

func main() {
  fmt.Printf("git hash: %s, build serial: %s", gitCommit, buildSerial)
}

Two variables in this module gitCommit, and buildSerial are going to hold some version information for us. Running this program yields some rather uninteresting results.

go run main.go
git hash: , build serial: 

-X switch

While building a program, you can use the -X linker switch which will allow you to supply information into module variables from the build process.

We can obtain the latest build hash using git with the following:

git rev-list -1 HEAD

We can even synthesize a build number involving the date, perhaps?

date +%s

Using the -ldflags switch, we can now specify these at the console.

go build -ldflags "-X main.gitCommit=`git rev-list -1 HEAD` -X main.buildSerial=`date +%s`" main.go

Closing up

Now that we have a binary built, it’s had its build information applied - these variables now magically receive these values.

./main
git hash: f6f62d9a759a03afffb913a1d24fb64a1bc5507d, build serial: 1564573076

Remember, these switches can be buried behid a Makefile also, so you don’t need to be typing these things over and over.

GOOS=linux
GOARCH=386

.PHONY: build

GIT_COMMIT := $(shell git rev-list -1 HEAD)
BUILD_NO := $(shell date +%s)

build:
  GOOS=$(GOOS) GOARCH=$(GOARCH) go build -ldflags "-X main.gitCommit=$(GIT_COMMIT) -X main.buildSerial=$(BUILD_NO)" .