Textfile processing with bash

10 Feb 2015

Today’s post will be a bash on liner that allows you to process text files, line-by-line.

while read line; do wget $line; done < urls.txt

In this case, urls.txt is a list of urls that we want to process using wget. $line is the value that we’re currently processing (the line of text).

This is a pretty heavy handed way of downloading a handful of URLs. You could much easier do it with xargs which I’d mentioned in a previous post. Anyway, the while way gives you a bit more room to breathe in your loops.

Developing with Tomcat user instances

10 Feb 2015

In today’s post, I’ll take you through installing Apache Tomcat for the purposes of development.

Installing a user instance

To get started, install tomcat8 as you normally would:

$ sudo apt-get install tomcat8

For the purposes of development, it makes sense to have your own instance of tomcat which is away from the system installation. The tomcat8-user package allows you to do just that.

$ sudo apt-get install tomcat8-user

After this install is complete, you can create yourself a local tomcat instance that you can blow up without hurting the system’s version. You do this with the tomcat8-instance-create command:

$ tomcat7-instance-create -p 10080 -c 10005 tomcat

The switches -p puts this instance listening for application requests on port 10080 and the -c switch puts the control port on 10005.

After you’ve done this, you’ll be notified by the console.

You are about to create a Tomcat instance in directory 'tomcat'
* New Tomcat instance created in tomcat
* You might want to edit default configuration in tomcat/conf
* Run tomcat/bin/startup.sh to start your Tomcat instance

The directory that has been setup for you now looks like this:

.
├── bin
│   ├── setenv.sh
│   ├── shutdown.sh
│   └── startup.sh
├── conf
│   ├── catalina.properties
│   ├── context.xml
│   ├── logging.properties
│   ├── server.xml
│   ├── tomcat-users.xml
│   └── web.xml
├── logs
├── temp
├── webapps
└── work

Integrating with Eclipse

To get Eclipse to play nicely with your user-local version of tomcat, you’ll still need to add a few components. This tip is largly based off of the information in this stack overflow question.

$ ln -s /usr/share/tomcat8/lib
$ cp /etc/tomcat8/policy.d/03catalina.policy conf/catalina.policy
$ ln -s /usr/share/tomcat8/bin/bootstrap.jar bin/bootstrap.jar
$ ln -s /usr/share/tomcat8/bin/tomcat-juli.jar bin/tomcat-juli.jar
$ mkdir -p common/classes;
$ mkdir -p server/classes;
$ mkdir -p shared/classes;

You can now add your local user instance of tomcat to Eclipse.

Start logging with log4j2

08 Feb 2015

Probably the easiest way to get logging into your java application is with log4j from Apache. In today’s post I’m going to setup some basic logging with the 2.x series of this library.

Getting installed

The first thing to do is to download log4j from Apache’s downloads page. Once you have the binary distribution, extract it out into your preferable location. I personally put all of my libraries under ~/src/lib that I use for my projects.

Add the following jars to your lib folder in your project that you’re going to add logging to:

log4j-api-2.1.jar
log4j-core-2.1.jar

The versions may be differ, but as long as you’ve referenced the api file and the core file, you’ll be fine.

First logs

In order to get started, you’ll need a concrete implementation of the Logger interface. This is pretty easy with the help of the LogManager class:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class LoggerTest {

static Logger log = LogManager.getLogger(
LoggerTest.class.getName()
);

public static void main(String[] args) {
}

}

We give getLogger the name of the class that we’re in so that the resulting log can reference this name.

From here on, you’ll reference the variable log to send anything into the log. Running this application as it is results in the following error from the log4j2 framework:

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

That’s fine, we can still take a look at what an error looks like then by adding the following line to main:

log.log(Level.ERROR, "Error message");

The result is that our line gets logged out:

16:25:16.656 [main] ERROR org.keystoreplay.LoggerTest - Error message

Configuration

The log4j manual has a whole section dedicated to configuring the framework. If you’re looking to do more complex things, that’s the best place to look.

Quoted directly from the manual are the following bullet points regarding the process that the framework will go through to perform automatic configuration:

Log4j will inspect the “log4j.configurationFile” system property and, if set, will attempt to load the configuration using the ConfigurationFactory that matches the file extension.
If no system property is set the YAML ConfigurationFactory will look for log4j2-test.yaml or log4j2-test.yml in the classpath.
If no such file is found the JSON ConfigurationFactory will look for log4j2-test.json or log4j2-test.jsn in the classpath.
If no such file is found the XML ConfigurationFactory will look for log4j2-test.xml in the classpath.
If a test file cannot be located the YAML ConfigurationFactory will look for log4j2.yaml or log4j2.yml on the classpath.
If a YAML file cannot be located the JSON ConfigurationFactory will look for log4j2.json or log4j2.jsn on the classpath.
If a JSON file cannot be located the XML ConfigurationFactory will try to locate log4j2.xml on the classpath.
If no configuration file could be located the DefaultConfiguration will be used. This will cause logging output to go to the console.

What I like to do is contain all of my configuration files under a directory; normally called etc or conf. It doesn’t matter where your configuration files go, just as long as the classpath references them.

A simple configuration for this system is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
    </Console>
  </Appenders>
  <Loggers>
    <Root level="trace">
      <AppenderRef ref="Console"/>
    </Root>
  </Loggers>
</Configuration>

This was based off a config block found in the configuration manual. Note that the Level attribute has been set to trace. This will throw everything to the console for us. Adding the following line:

log.log(Level.WARN, "A simple warning");

We get the following output (without an error message telling us that we haven’t configured to logger):

16:40:14.207 [main] WARN  org.keystoreplay.LoggerTest - A simple warning
16:40:14.210 [main] ERROR org.keystoreplay.LoggerTest - Error message

Keytool

08 Feb 2015

The Java Keystore is a file that contains your security certificates and keys. It’s a convenient way to ship security information around with your application, but requires a little administration work to have one built.

The KeyStore java class natively works with this technology so that your security information can be easily used inside of your applications.

In today’s post, I’ll go through some basic usage of the keytool application. There are so many more features to this application that what I’ve listed below, so check out the man page for keytool as a reference.

Creating a keystore and client requests

Create a keystore with a new key pair

$ keytool -genkey -alias mydomain -keyalg RSA -keystore keystore.jks -storepass password

This creates a key store and puts a key pair in it (based on the subject details that you provided). You can verify that the key pair is in the store by listing it out:

$ keytool -list -keystore keystore.jks

You should end up with output like the following

Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

mydomain, 08/02/2015, PrivateKeyEntry, 
Certificate fingerprint (SHA1): 0F:42:6D:F6:48:85:99:4C:B5:97:0B:25:10:BF:83:F9:D5:2A:80:77

The text PrivateKeyEntry tells us the particular entry contains a secret/private key.

keytool can also generate certificate signing requests from this created keystore now:

$ keytool -certreq -alias mydomain -keystore keystore.jks -storepass password -file mydomain.csr

The file mydomain.csr now contains the certificate request block.

In cases where you aren’t going to a certificate authority and you just want to generate a self-signed certificate, you can just do the following:

$ keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 365

This puts a self-signed certificate, valid for 1 year into the same store.

If you’ve imported a secret into your keystore that you’d like to change the password on, you can do the following:

Take note! This isn’t changing the keystore’s password. This is changing the private key’s password.

$ keytool -keypasswd -alias mydomain -keypass secret -new new_secret_password -keystore keystore.jks -storepass password

If you have a PKCS 12 (sometimes referred to as PFX), you can create a keystore with the key information using the following:

$ keytool -importkeystore -srckeystore keyfile.pfx -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS

Importing and exporting certificates

If you need to trust an intermediate or root certificate, you can import them like so:

$ keytool -import -trustcacerts -alias visa -file Visa_eCommerce_Root.crt -keystore keystore.jks -storepass password

Taking a look at how this entry looks in the keystore:

visa, 08/02/2015, trustedCertEntry, 
Certificate fingerprint (SHA1): 70:17:9B:86:8C:00:A4:FA:60:91:52:22:3F:9F:3E:32:BD:E0:05:62

You see that this item doesn’t mention PrivateKeyEntry as there is no secret stored in this entry, it’s only the certificate (public key) so it lists as trustedCertEntry.

The visa certificate that I’d just imported can now be exported with the following command:

$ keytool -export -alias visa -file visa.crt -keystore keystore.jks -storepass password

Viewing certificate detail

You can view the details of any certificate that you have on your file system using keytool as well:

$ keytool -printcert -v -file visa.crt

The verbose output allows you to check all of the details in the certificate. You can perform this certificate printing process on any certificate inside of a keystore, as well. In this case though, you need to refer to the certificate by its alias:

$ keytool -list -v -keystore keystore.jks -storepass password -alias visa

You’ll end up with identical output.

Other utilities

Finally, you can remove certificates from a keystore. Again, you need to reference the certificate by its alias:

$ keytool -delete -alias visa -keystore keystore.jks -storepass password

You can change the password for a keystore as well:

$ keytool -storepasswd -new my_new_password -keystore keystore.jks -storepass password

Drawing with Cairo

08 Feb 2015

Cairo is a cross platform 2D graphics library. It’s got a wide range of features that are exposed through a solid and easy to use API. From the website:

The cairo API provides operations similar to the drawing operators of PostScript and PDF. Operations in cairo including stroking and filling cubic Bézier splines, transforming and compositing translucent images, and antialiased text rendering. All drawing operations can be transformed by any affine transformation (scale, rotation, shear, etc.)

In today’s post, I’m going to re-implement a very simple version of the old windows screensaver, Mystify. In fact, it’s not even going to look as cool as the one in the video but it will take you through basic drawing with Cairo and animation using GTK+ and GDK.

Getting your feet wet

If you don’t want to dive right into doing animation with Cairo, I suggest that you take a look at the FAQ. Up there is a section on what a minimal C program looks like. For reference, I have included it below. You can see that it’s quite static in nature; writing a PNG of the result at the end:

#include <cairo.h>

int main (int argc, char *argv[]) {
  cairo_surface_t *surface =
     cairo_image_surface_create(
      CAIRO_FORMAT_ARGB32, 
      240, 
      80);

  cairo_t *cr =
     cairo_create(surface);

  cairo_select_font_face(
    cr, 
    "serif", 
    CAIRO_FONT_SLANT_NORMAL, 
    CAIRO_FONT_WEIGHT_BOLD
  );

  cairo_set_font_size(cr, 32.0);
  cairo_set_source_rgb(cr, 0.0, 0.0, 1.0);
  cairo_move_to(cr, 10.0, 50.0);
  cairo_show_text(cr, "Hello, world");

  cairo_destroy(cr);
  cairo_surface_write_to_png(
    surface, 
    "hello.png"
  );
  cairo_surface_destroy(surface);
  return 0;
}

Building Cairo applications

There’s a shopping list of compiler and linker switches when building with Cairo, GTK+ and GDK. pkg-config has been a great help here, so here are the CFLAGS and LFLAGS definitions from my Makefile:

CFLAGS := -g -Wall `pkg-config --cflags gtk+-3.0 gdk-3.0 cairo`
LFLAGS := `pkg-config --libs gtk+-3.0 gdk-3.0 cairo`

Setting up the UI

First job is to create a window that will host our drawing. This is all pretty standard boilerplate for any GTK+ application.

int main(int argc, char *argv[]) {
  GtkWidget *window;

  gtk_init(&argc, &argv);
  init_verts();

  window = gtk_window_new(
    GTK_WINDOW_TOPLEVEL
  );

  darea = gtk_drawing_area_new();
    gtk_container_add(
    GTK_CONTAINER(window), 
    darea
  );

  g_signal_connect(
    G_OBJECT(darea), 
    "draw", 
    G_CALLBACK(on_draw_event), 
    NULL
  );

  g_signal_connect(
    window, 
    "destroy", 
    G_CALLBACK(gtk_main_quit), 
    NULL
  );  

  gtk_window_set_position(
    GTK_WINDOW(window), 
    GTK_WIN_POS_CENTER
  );

  gtk_window_set_default_size(
    GTK_WINDOW(window), 
    WIN_WIDTH, 
    WIN_HEIGHT
  );

  gtk_window_set_title(
    GTK_WINDOW(window), 
    "Lines"
  );

  gtk_widget_show_all(window);

  (void)g_timeout_add(
    33, 
    (GSourceFunc)mystify_animate, 
    window
  );

  gtk_main();

  return 0;
}

The parts to really take note of here is the creation of our drawable, and it getting connected to the window:

darea = gtk_drawing_area_new();
gtk_container_add(
  GTK_CONTAINER(window), 
  darea
);

Attaching our custom draw function to the draw signal with g_signal_connect:

g_signal_connect(
  G_OBJECT(darea), 
  "draw", 
  G_CALLBACK(on_draw_event), 
  NULL
);

Setting up a timer with g_timeout_add to continually call the animation function:

(void)g_timeout_add(
  33, 
  (GSourceFunc)mystify_animate, 
  window
);

Drawing

The Mystify effect is just a handful of vertices bouncing around the screen with lines connecting them. Really quite simple and we can get away with a basic data structure to defined this:

struct tag_mystify_vert {
  int x, y;     /* x and y positions */
  int vx, vy;   /* x and y velocities */
  double r, g, b;  /* colour components */
};

typedef struct tag_mystify_vert mystify_vert;

mystify_vert verts[N_VERTS];

Drawing the structure is just enumerating over the array defined above and drawing the lines:

static gboolean on_draw_event(
GtkWidget *widget, 
cairo_t *cr, 
gpointer user_data) {
  int n;

  /* clear the background to black */
  cairo_set_source_rgb(cr, 0, 0, 0);
  cairo_paint(cr);

  /* draw lines between verts */
  for (n = 0; n < (N_VERTS - 1); n ++) {
    cairo_set_source_rgb(
      cr, 
      verts[n].r, 
      verts[n].g, 
      verts[n].b
    );

    cairo_move_to(
      cr, 
      verts[n].x, 
      verts[n].y
    );
    cairo_line_to(
      cr, 
      verts[n + 1].x, 
      verts[n + 1].y
    );

    cairo_set_line_width(cr, 1);
    cairo_stroke(cr);
  }

  /* draw a line between the first and last vert */
  n = N_VERTS - 1;

  cairo_set_source_rgb(
    cr, 
    verts[n].r, 
    verts[n].g, 
    verts[n].b
  );

  cairo_move_to(
    cr, 
    verts[n].x, 
    verts[n].y
  );

  cairo_line_to(
    cr, 
    verts[0].x, 
    verts[0].y
  );

  cairo_set_line_width(cr, 1);
  cairo_stroke(cr);

  return FALSE;
}

Pretty basic. We set our draw colour with cairo_set_source_rgb, cairo_move_to and cairo_line_to handle our start and end points for the lines. cairo_stroke makes the doughnuts.

Animating

Finally, we animate the structure. Nothing really of interest in this code block except for how we notify the UI that we need a redraw.

gboolean mystify_animate(GtkWidget *window) {
  animate_verts();

  gtk_widget_queue_draw_area(
  window, 
  0, 0, 
  WIN_WIDTH, WIN_HEIGHT
  );

  return TRUE;
}

gtk_widget_queue_draw_area invalidates the defined region and forces the redraw for us.

Putting it all together

The source for this most unexciting version of the Mystify effect can be found here.

Older Newer

Cogs and Levers A blog full of technical stuff

Textfile processing with bash

Developing with Tomcat user instances

Installing a user instance

Integrating with Eclipse

Start logging with log4j2

Getting installed

First logs

Configuration

Keytool

Creating a keystore and client requests

Importing and exporting certificates

Viewing certificate detail

Other utilities

Drawing with Cairo

Getting your feet wet

Building Cairo applications

Setting up the UI

Drawing

Animating

Putting it all together