Cogs and Levers A blog full of technical stuff

Network Traffic Analysis with tcpdump

30 Aug 2022

Introduction

Sometimes it can be of value to be able to isolate and analyse specific network traffic that is flowing through your network interface. tcpdump offers you this capability in a command line application.

There are many tutorials already that take you through tcpdump comprehensively, so this article will just be constrained to usages that have benefited me.

Reading the output

In order for this tool to be of any use, it pays to know how to read the output. In this example I’m capturing all of the port 80 traffic flowing through my network interface.

tcpdump -nnSX port 80

The stream of output that you see after this (once you have some port 80 traffic going) is the output that you’ll use for analysis. Here’s an excerpt after hitting the first page on the internet.

21:49:54.056071 IP 192.168.20.35.60548 > 188.184.21.108.80: Flags [P.], seq 3043086668:3043087150, ack 3119373143, win 502, options [nop,nop,TS val 3532078292 ecr 2432081556], length 482: HTTP: GET /hypertext/WWW/TheProject.html HTTP/1.1
        0x0000:  4500 0216 a4d4 4000 4006 ed1d c0a8 1423  E.....@.@......#
        0x0010:  bcb8 156c ec84 0050 b561 d14c b9ed db57  ...l...P.a.L...W
        0x0020:  8018 01f6 62a6 0000 0101 080a d287 3cd4  ....b.........<.
        0x0030:  90f6 9e94 4745 5420 2f68 7970 6572 7465  ....GET./hyperte
        0x0040:  7874 2f57 5757 2f54 6865 5072 6f6a 6563  xt/WWW/TheProjec
        0x0050:  742e 6874 6d6c 2048 5454 502f 312e 310d  t.html.HTTP/1.1.

There’s lots here.

We’re given the time of the packet being observed 21:49:54.056071.

We’re given the network layer protocol IP, source address (my machine) 192.168.20.35 and port 60584; along with the destination 188.184.21.108 (on port 80).

The next field Flags [P.] is an encoded representation of the TCP flags. The following table gives a breakdown of these flag values.

Value Flag Description
S SYN Connection start
F FIN Connection finish
P PUSH Data push
R RST Connection reset
. ACK Acknowledgement

The combination of values tells you the flags that are up. In this case P. tells us this is a PUSH-ACK packet.

The sequence number seq 3043086668:3043087150 tells us the run of bytes contained within this sample. The ack value ack 3119373143 is the next byte expected. The win value tells us the number of bytes available in the buffer followed by the TCP options.

The packet length is given at the end of the line.

The data frame is now split into a hexadecimal representation in the middle (given by -X); and the ASCII representation to the right.

With the basic output view out of the way, we get move onto some useful invocations.

Invocations

Filter by Port

As per the above example, we can filter traffic by any port that we give to port switch. Here we can see any SMTP traffic.

# just the traffic on port 25
tcpdump port 25

# traffic on ports ranging from 25 to 30
tcpdump portrange 25-30

Everything

Sometimes it can be useful to just receive everything flowing through a network interface.

tcpdump -i wlp4s0

Filter by Host

You can use the host keyword to see traffic going to or coming from an IP address. You can constrain this even further using src (coming from) or dest (going to).

tcpdump host 192.168.20.1

# packets going to 20.1
tcpdump dst 192.168.20.1

# packets coming from 20.1
tcpdump src 192.168.20.1

Filter by Network

Using broader strokes, you can use net to specify a full network to filter packets on. This will allow you to filter a whole network or subnet.

tcpdump net 192.168.20.0/24

Filter by Protocol

Just seeing ping (ICMP) traffic can be filtered like so:

tcpdump icmp

Conclusion

tcpdump is a very useful network analysis tool do perform discoveries on what’s actually happening. There’s a lot more power that can be unlocked by combining some of these basic filters together using logical concatenators.

MYO Language with Antlr

30 Aug 2022

Introduction

ANTLR is a code generation tool for making language parsers. Using a grammer file, you can get ANTLR to generate code to read, interpret, and execute your very own code.

In today’s article I’ll walk through the basic setup to create a Calculator language that can execute simple equations in a golang project of our own.

Before you begin

You’ll need a JRE.

Before we start, there are some software pre-requisites. You will need to install ANTLR. This is a simple JAR File that we can invoke locally.

$ wget http://www.antlr.org/download/antlr-4.7-complete.jar
$ alias antlr='java -jar $PWD/antlr-4.7-complete.jar'

Code generation

Now that we’ve got ANTLR installed, it’s time to generate some code. We do this using a grammer file. A very comprehensive calculator can be found in the examples of the antlr grammers repository here.

For today’s example, we’ll just focus on addition, subtraction, multiplication, and division with the following grammer file:

// Calc.g4
grammar Calc;

// Tokens
MUL: '*';
DIV: '/';
ADD: '+';
SUB: '-';
NUMBER: [0-9]+;
WHITESPACE: [ \r\n\t]+ -> skip;

// Rules
start : expression EOF;

expression
   : expression op=('*'|'/') expression # MulDiv
   | expression op=('+'|'-') expression # AddSub
   | NUMBER                             # Number
   ;

Even without fully understanding the grammer language, you can see that there is some basic token definitions, rules, and expression definitions.

MUL, DIV, ADD, SUB, NUMBER, and WHITESPACE all being significant to the language that we’re definting.

The expression definition not only defines operations for us, but will also be key in defining operator precedence, with the MulDiv rule occuring before the AddSub rule, finally dealing with Number.

We can turn this grammer file into some go code with the following invocation:

$ antlr -Dlanguage=Go -o parser Calc.g4

This creates a parser folder for us now with a few different pieces of go code.

Parsers, Lexers, and Listener

If you look in the parser folder at the code that was created, you shoul see something similar to this:

└── parser
    ├── calc_base_listener.go
    ├── calc_lexer.go
    ├── CalcLexer.tokens
    ├── calc_listener.go
    ├── calc_parser.go
    └── Calc.tokens

The Lexer’s job is to perform Lexical Analysis on arbitrary pieces of text, and tokenizes that text into a set of symbols. For example, the input of 1 + 2 might get tokenized to NUMBER 1, ADD, NUMBER 2. These tokens are now fed into the parser.

The Parser’s job is to take these tokens, and make sure they conform to the rules of the language. You can imagine that a LISP style language would expect ADD, NUMBER 1, NUMBER 2 rather than a c-style language that would expect the operator in between the number tokens.

After the string has passed through the lexer and the parser, it now runs through the listener where we can write some code to respond to these symbols in order.

Implementation

The internal implementation of this calculator is a stack-based calculator. This gets represented as struct:

type calculatorListener struct {
	*parser.BaseCalcListener
	stack []int
}

The internal state of the calculator are int values on that stack. As operations execute, the program will take the top of the stack as well that second-to-the-top and perform arithmetic, leaving the result on the top of the stack.

func (l *calculatorListener) push(i int) {
	l.stack = append(l.stack, i)
}

func (l *calculatorListener) pop() int {
	if len(l.stack) < 1 {
		panic("TOS invalid")
	}

	result := l.stack[len(l.stack)-1]
	l.stack = l.stack[:len(l.stack)-1]

	return result
}

The BaseCalcListner type that was generated for us has all of the hooks we need to latch onto the complete the implementation. The NUMBER, ADDSUB, and MULDIV rules all get their own listener for us to respond to.

func (l *calculatorListener) ExitMulDiv(c *parser.MulDivContext) {
  // get TOS and STOS
	rhs, lhs := l.pop(), l.pop()

  // perform the required operation, pushing the result back
  // up as the new TOS
	switch c.GetOp().GetTokenType() {
	case parser.CalcParserMUL:
		l.push(lhs * rhs)
	case parser.CalcParserDIV:
		l.push(lhs / rhs)
	default:
		panic(fmt.Sprintf("not yet implemented: %s", c.GetOp().GetText()))
	}
}

func (l *calculatorListener) ExitAddSub(c *parser.AddSubContext) {
  // get TOS and STOS
	rhs, lhs := l.pop(), l.pop()

  // perform the required operation, pushing the result back
  // up as the new TOS
	switch c.GetOp().GetTokenType() {
	case parser.CalcParserADD:
		l.push(lhs + rhs)
	case parser.CalcParserSUB:
		l.push(lhs - rhs)
	default:
		panic(fmt.Sprintf("not yet implemented: %s", c.GetOp().GetText()))
	}
}

func (l *calculatorListener) ExitNumber(c *parser.NumberContext) {
  // coerce the string into an integer
	i, err := strconv.Atoi(c.GetText())
	if err != nil {
		panic(err.Error())
	}

  // push onto the stack
	l.push(i)
}

Execution

Now we go from text input to execution. In the following snippet, the input stream feeds the text into the lexer. The lexer then gets setup as a stream ready to tokenize our input.

Finally, all of those tokens get parsed to make sure they represent valid expressions for our language.

equation := "1 + 5 - 2 * 20"
is := antlr.NewInputStream(equation)
lexer := parser.NewCalcLexer(is)
stream := antlr.NewCommonTokenStream(lexer, antlr.TokenDefaultChannel)

p := parser.NewCalcParser(stream)

We can now walk the parser tree with a listener attached. The listener will fire off our hooks that we defined earlier; and our stack-based calculator should leave us with the result at the TOS.

var listener calcListener
antlr.ParseTreeWalkerDefault.Walk(&listener, p.Start())
answer := listener.pop()

fmt.Printf("%s = %d", equation, answer)

We should be left with something like this on screen:

1 + 5 - 2 * 20 = -34

Conclusion

As you can see, ANTLR is a very powerful tool for writing all of the pieces of a compiler (or in this case, an interpreter) to get you kick started very quickly.

You’d almost be insane to ever do this stuff yourself!

Shell Tricks

15 Feb 2022

Sometimes you can be just as productive using your shell as you are in any programming environment, you just need to know a couple of tricks. In this article, I’ll walk through some basic tips that I’ve come across.

Reading input

You can make your scripts immediately interactive by using the read instruction.

#/bin/bash

echo -n "What is your name? "
read NAME

echo "Hi there ${NAME}!"

String length

You can get the length of any string that you’ve stored in a variable by prefixing it with #.

#/bin/bash

echo -n "What is your name? "
read NAME

echo "Your name has ${#NAME} characters in it"

Quick arithmetic

You can perform some basic arithmetic within your scripts as well. The value emitted with the # character is an integral value that we can perform tests against.

#/bin/bash

echo -n "What is your name? "
read NAME

if (( ${#NAME} > 10 )) 
then
  echo "You have a very long name, ${NAME}"
fi

Substrings

String enumeration will also allow you to take a substring directly. The format takes the form of ${VAR:offset:length}.

Passing positive integers for offset and length will make substring operate from the leftmost side of the string. Negative numbers provide a reverse index, from the right.

STR="Scripting for the win"

echo ${STR:10:3}
# for

echo ${STR: -3}
# win

echo ${STR: -7: 3}
# the

Replacement

It’s common place to be able to use regular expressions to make substitutions where needed, and they’re available to you at the shell as well.

#!/bin/bash

STR="Scripting for the win"

echo ${STR/win/WIN}
# Scripting for the WIN

Finishing up

There’s lots more that you can do just from the shell, without needing to reach for other tools. This is only a few tips and tricks.

Mutual TLS (mTLS)

27 Dec 2020

Introduction

TLS has forever played a very large part in securing internet communications. Secure Socket Layer (SSL) filled this space prior to TLS coming to the fore.

In today’s article, I’m going to walk through an exercise of mTLS which is just an extension of TLS.

CA

First of all, we need a certificate authority (CA) that both the client and the server will trust. We generate these using openssl.

openssl req -new -x509 -nodes -days 365 -subj '/CN=my-ca' -keyout ca.key -out ca.crt

This now puts a private key in ca.key and a certificate in ca.crt on our filesystem. We can inspect these a little further with the following.

openssl x509 --in ca.crt -text --noout

Looking at the output, we see some interesting things about our CA certificate. Most importantly the X509v3 Basic Constraints value is set CA:TRUE, telling us that this certificate can be used to sign other certificates (like CA certificates can).

Server

The server now needs a key and certificate. Key generation is simple, as usual:

openssl genrsa -out server.key 2048

We need to create a certificate that has been signed by our CA. This means we need to generate a certificate signing request, which is then used to produce the signed certificate.

openssl req -new -key server.key -subj '/CN=localhost' -out server.csr

This gives us a signing request for the domain of localhost as mentioned in the -subj parameter. This signing request now gets used by the CA to generate the certificate.

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out server.crt

Inspecting the server certificate, you can see that it’s quite a bit simpler than the CA certificate. We’re only able to use this certificate for the subject that we nominated; localhost.

Client

The generation of the client certificates is very much the same as the server.

# create a key
openssl genrsa -out client.key 2048

# generate a signing certificate
openssl req -new -key client.key -subj '/CN=my-client' -out client.csr

# create a certificate signed by the CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out client.crt

The subject in this case is my-client.

The -CAcreateserial number also ensures that we have unique serial numbers between the server and client certificates. Again, this can be verified when you inspect the certificate.

# Server serial number
        Serial Number:
            5c:2c:47:44:2c:13:3b:c9:56:56:99:37:3f:c9:1e:62:c4:c7:df:20

# Client serial number
        Serial Number:
            5c:2c:47:44:2c:13:3b:c9:56:56:99:37:3f:c9:1e:62:c4:c7:df:21

Only the last segment was incremented here. You get the idea though. Unique.

Appliation

Now, we setup a basic node.js server that requires mTLS.

const https = require('https');
const fs = require('fs');

const hostname = 'localhost';
const port = 3000;

const options = { 
    ca: fs.readFileSync('ca.crt'), 
    cert: fs.readFileSync('server.crt'), 
    key: fs.readFileSync('server.key'), 
    rejectUnauthorized: true,
    requestCert: true, 
}; 

const server = https.createServer(options, (req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Most important here is that the server’s options specify rejectUnauthorized as well as requestCert. This will force the mTLS feedback look back to the client.

A curl request now verifies that the solution is secured by this system of certificates.

curl --cacert ca.crt --key client.key --cert client.crt https://localhost:3000

The client’s key, certificate, and the ca cert accompany a successful request. A request in any other format simply fails as the authentication requirements have not been met.

Find the listening process for a port

22 Dec 2020

Introduction

In networking, a port is assigned as a logical entity that a socket is established on. These sockets are owned by processes in your operation system. From time to time, it can be unclear which process owns which socket (or who is hogging which port).

In today’s article, I’ll take you through a few techniques on finding out who is hanging onto particular ports.

netstat

netstat is a general purpose network utility that will tell you about activity within your network interfaces.

netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

If you can not find netstat installed on your system, you can normally get it from the net-tools package.

The following command will give you a breakdown of processes listening on port 8080, as an example:

➜  netstat -ltnp | grep -w ':8080'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::8080                 :::*                    LISTEN      -

An important message appears here. “Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.”. There will be processes invisible to you unless you run this command as root.

Breaking down the netstat invocation:

  • l will only show listening sockets
  • t will only show tcp connections
  • n will show numerical addresses
  • p will show you the PID

You can see above, that no process is shown. Re-running this command as root:

➜  sudo netstat -ltnp | grep -w ':8080'
tcp6       0      0 :::8080                 :::*                    LISTEN      2765/docker-proxy

lsof

lsof will give you a list of open files on the system. Remember, sockets are just files. By using -i we can filter the list down to those that match on an internet address.

➜  sudo lsof -i :8080
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 2765 root    4u  IPv6  36404      0t0  TCP *:http-alt (LISTEN)

fuser

fuser is a program that has overlapping responsibilities with the likes of lsof.

fuser — list process IDs of all processes that have one or more files open

You can filter the list down directly with the command:

➜  sudo fuser 8080/tcp
8080/tcp:             2765

This gives us a PID to work with. Again, note this is run as root. Now all we need to do is to tranform this PID into a process name. We can use ps to finish the job.

➜  ps -p 2765 -o comm=
docker-proxy