DROP DATABASE and other users

From time to time, when I’ve gone and issued DROP DATABASE on my postgres server, I’m returned with the following error:

ERROR:  database "xyz" is being accessed by other users
DETAIL:  There is 1 other session using the database.

All this is telling us is that we need to be the only user/connection on the database before performing such an operation.

First of all, we need to prevent other users from making a connection to the database from this point. To do this, we’ll use REVOKE CONNECT like so:

REVOKE CONNECT ON DATABASE "xyz" FROM public;

Next, we kill off every connection:

SELECT pg_stat_activity.pid, pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'xyz';

Now you can issue your DROP DATABASE.

Detailed GC logs

From time to time, it makes sense to perform some GC tuning on your Java Virtual Machines. Whilst there are a lot of tools that can visually help your debugging process, in today’s post I’ll talk you through the GC log that you can optionally turn on in your virtual machine arguments.

Enabling the log

To boost up the logging of your application, you’ll need to tune the execution runtime using command line parameters. The following parameters will get the JVM to log out information that it’s holding on garbage collection events.

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc.log

To explain these a little:

-verbose:gc will ramp the logging level of GC events up to a verbose level, -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps define some features of the log that’s written. Finally -Xloggc:/tmp/gc.log defines the file endpoint on disk that the GC log will be written to.

Reading the log

After you’ve run your program with these parameters engaged, you should find the /tmp/gc.log file sitting on your hard drive waiting to be read. I won’t dump the full log for the test program that I’ve run here; rather I’ll go through it piece by piece.

The header of the file defines what your software versions, memory statistics and virtual machine arguments are.

OpenJDK 64-Bit Server VM (25.66-b01) for linux-amd64 JRE (1.8.0_66-internal-b01), built on Aug  5 2015 09:09:16 by "pbuilder" with gcc 4.9.2
Memory: 4k page, physical 8055396k(6008468k free), swap 8267772k(8267772k free)
CommandLine flags: -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC 

After these initial lines, you’ll start to see some of the memory allocation events appear along with the timestamps (remember, we asked for timestamps above).

0.320: [GC (Allocation Failure) [PSYoungGen: 262144K->43488K(305664K)] 262144K->142896K(1005056K), 0.1552445 secs] [Times: user=0.40 sys=0.20, real=0.16 secs] 
0.649: [GC (Allocation Failure) [PSYoungGen: 305632K->43504K(305664K)] 405040K->275088K(1005056K), 0.2105517 secs] [Times: user=0.58 sys=0.26, real=0.21 secs] 
0.986: [GC (System.gc()) [PSYoungGen: 219445K->43520K(305664K)] 451029K->369480K(1005056K), 0.1570988 secs] [Times: user=0.47 sys=0.14, real=0.15 secs] 
1.143: [Full GC (System.gc()) [PSYoungGen: 43520K->0K(305664K)] [ParOldGen: 325960K->368132K(699392K)] 369480K->368132K(1005056K), [Metaspace: 2530K->2530K(1056768K)], 2.5983336 secs] [Times: user=9.55 sys=0.03, real=2.59 secs] 
3.984: [GC (Allocation Failure) [PSYoungGen: 262144K->32K(305664K)] 630276K->368164K(1005056K), 0.0049817 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
4.070: [GC (System.gc()) [PSYoungGen: 108791K->32K(305664K)] 476924K->368164K(1005056K), 0.0041558 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
4.074: [Full GC (System.gc()) [PSYoungGen: 32K->0K(305664K)] [ParOldGen: 368132K->133835K(699392K)] 368164K->133835K(1005056K), [Metaspace: 2539K->2539K(1056768K)], 0.4427402 secs] [Times: user=1.59 sys=0.01, real=0.45 secs] 

Pulling one of these lines apart:

0.320: [GC (Allocation Failure) [PSYoungGen: 262144K->43488K(305664K)] 262144K->142896K(1005056K), 0.1552445 secs] [Times: user=0.40 sys=0.20, real=0.16 secs] 

This event was generated 0.320 seconds into the program. This item is a GC (Allocation Failure) event and it’s being reported on the PSYoungGen collection. Prior to the event, the space allocated before was 262144K and after was 43488K. The capacity value is in braces 305664K.

The Full GC events will give you statistics for all of the memory collections:

1.143: [Full GC (System.gc()) [PSYoungGen: 43520K->0K(305664K)] [ParOldGen: 325960K->368132K(699392K)] 369480K->368132K(1005056K), [Metaspace: 2530K->2530K(1056768K)], 2.5983336 secs] [Times: user=9.55 sys=0.03, real=2.59 secs] 

Each of the collections is displayed as [CollectionName: SpaceBefore->SpaceAfter(Capacity)].

Finally, we have a heap analysis of the program as it breaks down amongst the different memory classes: Young Gen, Old Gen and (new for 1.8) Metaspace. Metaspace would have previously been Perm Gen.

Heap
 PSYoungGen      total 305664K, used 5243K [0x00000000eab00000, 0x0000000100000000, 0x0000000100000000)
  eden space 262144K, 2% used [0x00000000eab00000,0x00000000eb01ecf8,0x00000000fab00000)
  from space 43520K, 0% used [0x00000000fab00000,0x00000000fab00000,0x00000000fd580000)
  to   space 43520K, 0% used [0x00000000fd580000,0x00000000fd580000,0x0000000100000000)
 ParOldGen       total 699392K, used 133835K [0x00000000c0000000, 0x00000000eab00000, 0x00000000eab00000)
  object space 699392K, 19% used [0x00000000c0000000,0x00000000c82b2c88,0x00000000eab00000)
 Metaspace       used 2546K, capacity 4486K, committed 4864K, reserved 1056768K
  class space    used 268K, capacity 386K, committed 512K, reserved 1048576K

Standard object protocols in Python

To give your objects a more baked-in feel, you can use python’s standard object protocol functions so that native operators start to operate on your object.

By implementing the following items on your custom objects, infix operators start to work executing your custom code as per defined.

General

By overriding __bool__ in your objects, you can define how your object will respond in conditional scenarios. __bool__ effectively allows you to use your object as a condition in an if or while statement.

class Boolable:
  def __init__(self, n):
    self.num = n

  def __bool__(self):
    return self.num % 2 == 0    

The method __call__ will allow your object to openly accept function calls:

class Callable:
  def __call__(self, *args, *kwargs):
    # implementation here

c = Callable()
c()

Array

The following overrides allow you to make your objects appear like containers (arrays, etc.):

Dictionary

The following overrides allow you to make your object respond like a dict:

Mathematic

The following table lists out all of the methods that you can override on a class that will give you access to arithmetic operators.

Comparison

The following table lists all of the comparison operators

Type conversions

Context

The following override allow your objects to measure contexts:

These functions are useful when your object is supplied to a with statement.

class ContextMeasurement:
  def __enter__(self):
    print("Entering context")
      
  def __exit__(self, exc_class, exc_instance, traceback):
    print("Exiting context")
        
with ContextMeasurement():
  print("Inside the context right now")

Getting started with Akka

Akka is a library designed for building applications using the actor model. From their site:

Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM.

In today’s post, I’m going to start with some of the primitives to using this framework.

Messages

Actors process messages that you’ll define in your modules. For today’s example, I’m going to implement a very basic logging application. Messages sent into this system are expected to be logged out to the console. To start off, we define the messages for this system:

case object Log
case class LogMessage(when: Date, level: String, text: String)
case class LogString(message: String)
case class LogException(e: Exception)

Using scala’s case classes we can clean up the definition of these log messages. We have a message that will do general logging LogMessage, one that will log a string in LogString and one that will dissect and log out an exception object LogException.

Actor Logic

We now focus on the logic required to log information out from our actor. This is really quite simple; we’re just going to push everything out to the console:

class LogActor extends Actor {

  def receive = {
    case LogMessage(when, level, text) => println(String.format("%s [%s] %s", when.toString(), level, text))
    case LogString(message) => self ! LogMessage(new Date, "info", message)
    case LogException(e) => self ! LogMessage(new Date, "error", e.toString())
  }

}

The receive method is just a big pattern matching statement. Each of the message types are handled in here. Note how LogString and LogException send messages to self. self is a built-in, given to us representing this actor. All we’re doing is just on-forwarding the message in the string and exception cases.

Creating a system

We have actors; we have messages to pass between the actors; we now need a system that the actors will participate in.

// create the system
val system = ActorSystem("myLoggingSystem")

// create an actor
val logger = system.actorOf(Props[LogActor], "logger")

Using the tell and ask methods, we can send and send/receive messages to/from this actor. We also can create a logic-less actor that just acts as a message sender/receiver:

val inbox = Inbox.create(system)

Mailboxes are an important abstraction; they hold messages for actors. Each actor has its own mailbox, but we’ve created one above attached to a system that we can pipe messages into:

inbox.send(logger, LogString("This is the first line of log"))
inbox.send(logger, LogException(new Exception("DOH!")))

Lots of Actors

A slightly more complex topic is to create a pool of actors. In this next snippet, we’ll create a RoundRobinPool.

val actors = system.actorOf(Props[LogActor].withRouter(RoundRobinPool(5)), name = "LoggingActors")

Now that we’ve created a pool, it’s time to smash!

Range(1, 1000000).map(i => actors ! LogString(String.format("Message number %s", i.toString())))

Scheduled

Finally, we can schedule these messages to be sent . . as if they were sent from no where using the actor system that we’d created earlier:

system.scheduler.schedule(0.seconds, 1.second, actors, LogString("Yerrr!"))(system.dispatcher, Actor.noSender)

This will send a LogString message to the actor system actors after zero seconds and then another message every second there after.

Bash Gems

The Bourne Again SHell is one of the most widely deployed shell for Linux that I use all the time. In today’s post, I’m going to collate a lot of the gems that I’d discovered in my travels of using this software.

Finding Help

Nothing can substitute the reference manual materials distributed with this software when it’s installed. At the console, you can read documentation in info format on bash using the following:

info bash

You’re able to deduce executing this command by doing some research at the console, by yourself. Using apropos (which searches the manual pages) you can look for key words.

If you wanted to find any command that begins with the characters ‘ls’ in an attempt to find the command ls, you can perform the following search:

apropos ls | grep '^ls.*'

On my system here, this function emits the following result:

ls (1)               - list directory contents
lsattr (1)           - list file attributes on a Linux second extended file s...
lsb_release (1)      - print distribution-specific information
lsblk (8)            - list block devices
lscpu (1)            - display information about the CPU architecture
lsdiff (1)           - show which files are modified by a patch
lsearch (3)          - linear search of an array
lseek (2)            - reposition read/write file offset
lseek64 (3)          - reposition 64-bit read/write file offset
lshw (1)             - list hardware
lsinitramfs (8)      - list content of an initramfs image
lslocks (8)          - list local system locks
lsmod (8)            - Show the status of modules in the Linux Kernel
lsof (8)             - list open files
lspci (8)            - list all PCI devices
lspcmcia (8)         - display extended PCMCIA debugging information
lspgpot (1)          - extracts the ownertrust values from PGP keyrings and l...
lstat (2)            - get file status
lstat64 (2)          - get file status
lsusb (8)            - list USB devices

We’re only interested in the first item there, but we’re given all of the options. We can now display the manual page with the following:

man ls 1

Variables

Variable creation is fairly straight forward:

# stores the string "John" in var1
var1="John"
# stores the text output of the command 'ls' into var2
var2=`ls -al`

# simple string replacement
var3=${var1/h/a}
# sub-string (turns "John" into "Jo")
var4=${var1:0:2}
# default string substitution (where null)
var6=${var5:-"Value for var5 was not supplied"}

# string interpolation is achieved with $
echo "His name is $var1"

Special variables exist to tell the developer a little bit about their environment:

Functions

# define a function
function syntax() {
  echo "usage: prog.sh [options]"
  return 0
}

function print_name() {
  echo "Hello $1"
  return 0
}

# call the function
syntax
print_name "John"

Control Flow Constructs

# Conditionals
if [ var1 == 10 ] then
  echo "It was 10"
else
  echo "It was not 10"
fi

case "$var1" in
  0) echo "Value was zero";;
  1) echo "Value was one";;
  *) echo "Anything but null";;
esac 

# Repetition
for var1 in {1..10} do
done

for ((x=1; x <= 10; x++)) do
done

while [ var1 == 10 ] do
done

Redirection

Special file descriptors of 0 as /dev/stdin, 1 as /dev/stdout and 2 as /dev/stderr.

From the redirections section in the bash manual:

Note that the order of redirections is significant. For example, the command

ls > dirlist 2>&1

directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command

ls 2>&1 > dirlist

directs only the standard output to file dirlist, because the standard error was made a copy of the standard output before the standard output was redirected to dirlist.