Cogs and Levers A blog full of technical stuff

Convenience and performance with Python collections

17 Jan 2016

The collections library included with python has some very helpful utilities to make your programming life a little easier. In today’s post, I’m going to go through a few of them.

Named tuple

This is really the feature that brought my attention to this library, initially. Where a tuple is an immutable set of values that are unnamed, you can create a class using the namedtuple() function to bring a little more formality to your types:

import collections
Person = collections.namedtuple('Person', ['firstName', 'lastName', 'age'])
joe = Person("Joe", "Smith", 21)

# joe is now as follows
Person(firstName='Joe', lastName='Smith', age=21)

That’s a neat shortcut.

Counter

A Counter class is a dict object that when queried for a key that doesn’t exist, will return a 0; and create that item ready for counting.

animals = ['dog', 'cat', 'cat', 'bat', 'mouse', 'dog', 'elephant']
c = collections.Counter()

for animal in animals:
    c[animal] += 1

# "c" now looks like this
# Counter({'dog': 2, 'cat': 2, 'bat': 1, 'elephant': 1, 'mouse': 1})

Pretty handy.

deque

A basic stack or queue like data structure can be initialized with the use of the deque class. As this object does look a lot like a list, it’s important to remember why it exists:

Though list objects support similar operations, they are optimized for fast fixed-length operations and incur O(n) memory movement costs for pop(0) and insert(0, v) operations which change both the size and position of the underlying data representation.

This tells us that some internal implementation assumptions have been made to tailor the runtime usecase of this class specifically for statically sized queues.

orders = collections.deque()
orders.append({ 'name': 'Mario', 'pizza': 'Cheese' })
orders.append({ 'name': 'Joe', 'pizza': 'Supreme' })
orders.append({ 'name': 'Tony', 'pizza': 'Pepperoni' })

orders.popleft()
# {'name': 'Mario', 'pizza': 'Cheese'}

orders.popleft()
# {'name': 'Joe', 'pizza': 'Supreme'}

orders.popleft()
# {'name': 'Tony', 'pizza': 'Pepperoni'}

Scala

16 Jan 2016

Introduction

The Scala Programming Language is a language that brings together object oriented concepts with functional programming concepts on top of the jvm.

In today’s post, I’m going to go through some basic concepts of the language.

Classes

For our example, we’ll define a Player class. This will hold our player’s name, height and weight which won’t change once we set them. Note that we’re using the val keyword in the parameter list of the default constructor for the class. This automatically generates immutable members for us accessing this information.

class Player(val name: String, val height: Int, val weight: Int) {

  def getMessage(): String = "Game on!"

  def talk(): Unit = {
    val message = this.getMessage()
    println(s"$name the player says '$message'")
  }

}

We’ve also given our player the ability to talk. The player also has a message to say with getMessage.

Inheritance

We can inherit from this base Player class and define a Forward and a Back.

class Forward(name: String, height: Int, weight: Int) extends Player(name, height, weight) {
  override def getMessage(): String = "Uggg!"
}

class Back(name: String, height: Int, weight: Int) extends Player(name, height, weight) {
  override def getMessage(): String = "How does my hair look?"
}

Forwards and backs say different things, so we have overridden the default getMessage implementation in each case.

Traits

A trait is similar to the interface that you’d find in other languages. The main difference to a strict interface, is that a trait can have implementation. In the following example, the ValueEmitter trait is applied to different types of objects, but it commonly utilised to equate an answer.

trait ValueEmitter {
  def value(): Double
}

To represent a literal value and an operation both using this trait, we apply it to classes:

class LiteralValue(v: Double) extends ValueEmitter {
  def value(): Double = v
}

class Operation(val v1: ValueEmitter, val v2: ValueEmitter, val op: String) extends ValueEmitter {
  def value(): Double = {
    val left = v1.value()
    val right = v2.value()

    op match {
      case "+" => left + right
      case "-" => left - right
      case "*" => left * right
      case "/" => left / right
      case default => 0
    }
  }
}

Case Classes

Case classes allow you to concisely condense the definitions above:

abstract class ValueEmitter
case class LiteralValue(v: Double) extends ValueEmitter
case class Operation(v1: ValueEmitter, v2: ValueEmitter, op: String) extends ValueEmitter

This syncs up really well with the pattern matching ideas.

Pattern Matching

Following on with the example in Case Classes, we’ll write a function that uses pattern matching to ensure we’re getting the correct type through. Also see that we can pattern match on the values being passed through; not just the type.

def calculate(v: ValueEmitter): Double = v match {
  case LiteralValue(lv) => lv
  case Operation(v1, v2, "/") => {
    throw new Exception("I do not support divide")
  }
  case Operation(v1, v2, op) => {
    val left = calculate(v1)
    val right = calculate(v2)

    op match {
      case "+" => left + right
      case "-" => left - right
      case "*" => left * right
      case "/" => left / right
      case default => 0
    }

  }
}

Object

Static classes (or singleton objects) are just a way of defining single-use classes. You’ll see main sitting within an object definition rather than seeing main declared statically.

object Test {
  def main(args: Array[String]): Unit = {
    println("Hello, world")
  }
}

Another demonstrative use case for these objects is a configuration class:

object Config {
  def transactionDb(): String = "postgres://blah/xyz"
  def objectStoreDb(): String = "mongodb://quxx/abc"
}

Both transactionDb and objectStoreDb become accessible when prefixed with Config..

Accessors

You can short cut the creation of your accessors using your default constructor. As you’d expect, you use val for immutable, read-only properties and var for the read/write items.

class StockPrice(val code: String, var price: Double) {
}

The code on the stock doesn’t change but its price does.

These accessors can be defined manually using the following convention; this also allows you to specify any code that needs to execute within these accessors:

class StockPrice(val code: String, val initialPrice: Double) {

  private var _price: Double = initialPrice;

  def price: Double = _price
  def price_= (value: Double): Unit = {
    _price = value
  }

}

This is only a really small sample of the scala language, but will certainly get you up and running pretty quickly.

For more information, see the following links:

information_schema with PostgreSQL

10 Jan 2016

The information_schema in PostgreSQL holds a lot of really handy views with information about the current database. Very useful in investigation and discovery scenarios.

In today’s post, we’ll go through the tables that sit in this schema and how they can help. The information_schema documentation can be found here and is what this article has been based on.

Meta and Context

-- get the current database name
SELECT * FROM information_schema.information_schema_catalog_name;

-- what are the roles does the current user have that the admin option for?
SELECT * FROM information_schema.administrable_role_authorizations;

-- what roles are applicabl to the current user?
SELECT * FROM information_schema.applicable_roles;

-- attributes on composite data types that the current user has access to
SELECT * FROM information_schema.attributes;

Server

-- available character sets
SELECT * FROM information_schema.character_sets;

-- list all collations available to this database
SELECT * FROM information_schema.collations;

-- lists the available character sets that apply to the collations
SELECT * FROM information_schema.collation_character_set_applicability;

-- list all of the options defined for foreign-data wrappers
SELECT * FROM information_schema.foreign_data_wrapper_options;

-- list all foreign-data wrappers defined in the database
SELECT * FROM information_schema.foreign_data_wrappers;

-- lists all of the options defined for foreign servers in this database
SELECT * FROM information_schema.foreign_server_options

-- lists all of the standard sql features supported
SELECT * FROM information_schema.sql_features;

-- lists features that are implementation defined
SELECT * FROM information_schema.sql_implementation_info;

-- lists all of the sql languages supported
SELECT * FROM information_schema.sql_languages;

-- lists all of the sql defined feature packages are supported
SELECT * FROM information_schema.sql_packages;

-- lists all of the supported parts of the sql standard
SELECT * FROM information_schema.sql_parts;

-- lists the size limits in the database
SELECT * FROM information_schema.sql_sizing;

-- lists sizing profile information
SELECT * FROM information_schema.sql_sizing_profiles;

-- lists all of the foreign servers defined in this database
SELECT * FROM information_schema.foreign_servers;

-- lists all of the options defined for foreign tables in this database
SELECT * FROM information_schema.foreign_table_options;

-- lists all of the foreign tables 
SELECT * FROM information_schema.foreign_tables;

-- list all settings for user mappings
SELECT * FROM information_schema.user_mapping_options;

-- list all user mappings 
SELECT * FROM information_schema.user_mappings;

Catalog

-- list all check constraints
SELECT * FROM information_schema.check_constraints;

-- lists all of the parameters to functions in the database
SELECT * FROM information_schema.parameters;

-- lists all foreign keys in the database
SELECT * FROM information_schema.referential_constraints;

-- lists all of the functions in the database
SELECT * FROM information_schema.routines;

-- lists all of the sequences
SELECT * FROM information_schema.sequences;

-- lists all constraints from tables in this database
SELECT * FROM information_schema.table_constraints;

-- list all tables
SELECT * FROM information_schema.tables;

-- list all triggers
SELECT * FROM information_schema.triggers;

-- list all composite types
SELECT * FROM information_schema.user_defined_types;

-- lists all views in the database
SELECT * FROM information_schema.views;

-- list all transforms (9.5 ONLY)
SELECT * FROM information_schema.transforms;

Security and Privileges

-- list all columns and their priviledges
SELECT * FROM information_schema.column_privileges;

-- lists all privileges on columns
SELECT * FROM information_schema.role_column_grants;

-- lists all privileges on functions
SELECT * FROM information_schema.role_routine_grants;

-- lists all privileges on tables
SELECT * FROM information_schema.role_table_grants;

-- lists all privileges on udfs
SELECT * FROM information_schema.role_udt_grants;

-- lists all privileges on various objects 
SELECT * FROM information_schema.role_usage_grants;

-- lists all privileges on functions
SELECT * FROM information_schema.routine_privileges;

-- lists all of the table privileges 
SELECT * FROM information_schema.table_privileges;

-- list all udt privileges
SELECT * FROM information_schema.udt_privileges;

-- list privileges on various objects
SELECT * FROM information_schema.usage_privileges;

-- list all data types that the user has access to
SELECT * FROM information_schema.data_type_privileges;

-- list all enabled roles
SELECT * FROM information_schema.enabled_roles;

Explore

-- list all routines that are used by a check constraint
SELECT * FROM information_schema.check_constraint_routine_usage;

-- list columns using a domain defined inside of this database
SELECT * FROM information_schema.column_domain_usage;

-- list all columns that use types owned by the current user
SELECT * FROM information_schema.column_udt_usage;

-- list all columns used by constraints
SELECT * FROM information_schema.constraint_column_usage;

-- list all tables used by constraints
SELECT * FROM information_schema.constraint_table_usage;

-- list all domains based on data types owned by the current user
SELECT * FROM information_schema.domain_udt_usage;

-- lists all columns in the database restricted by primary,unique, foreign or check constraint
SELECT * FROM information_schema.key_column_usage;

-- list all columns that are used in views
SELECT * FROM information_schema.view_column_usage;

-- list all routines that are used in views
SELECT * FROM information_schema.view_routine_usage;

-- lists all tables that are used in views
SELECT * FROM information_schema.view_table_usage;

-- list all of the columns in the database
SELECT * FROM information_schema.columns;

-- list all triggers that specify update columns
SELECT * FROM information_schema.triggered_update_columns;

-- list options for any foreign table columns
SELECT * FROM information_schema.column_options;

-- list all constraints that belong to domains in the current database
SELECT * FROM information_schema.domain_constraints;

-- list all domains defined in the database
SELECT * FROM information_schema.domains

-- list all of the data types inside of array elements
SELECT * FROM information_schema.element_types;

-- lists all of the schemas 
SELECT * FROM information_schema.schemata;

Running X11 applications with port forwarding

07 Dec 2015

Today’s post is a quick tip on X11 port forwarding, and how to use it to run X11 applications remotely.

The setup

Your remote computer, the one that will actually run the application needs openssh installed. Use your favorite package manager to get that installed. You then need to edit your sshd configuration file to allow X11 port forwarding.

sudo emacs /etc/ssh/sshd_config

You need to make two edits to this file:

X11Forwarding     yes
X11UseLocalhost   no

Restart the ssh daemon.

Running

From your client computer now, connect to your remote host and run any X11 application that you want. It’ll appear on your client machine.

ssh -XC user@host
/usr/bin/firefox

Create a web service with maven

05 Dec 2015

In today’s post, I’m going to walk through a simple SOAP web service creation using maven, jax-ws for java. The service will be hosted inside of Apache Tomcat once we’re up and running.

Maven

First off, we start the application off with maven.

$ mvn archetype:generate -DgroupId=org.test.ws -DartifaceId=soap-test

This creates our project structure and puts all of the project dependencies in place. The pom.xml that gets generated for us needs a little extra help for a JAX-WS project. We need to:

  • Set the packaging to war
  • Add the jaxws-rt dependencies
  • Supply a final name

Your pom.xml should look something like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>org.test.ws</groupId>
  <artifactId>soap-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>war</packaging>

  <name>soap-test</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.sun.xml.ws</groupId>
      <artifactId>jaxws-rt</artifactId>
      <version>2.2</version>
    </dependency>    
    <dependency>
      <groupId>com.sun.istack</groupId>
      <artifactId>istack-commons-runtime</artifactId>
      <version>2.22</version>
    </dependency>    
  </dependencies>

  <build>
    <finalName>HelloService</finalName>
  
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <encoding>UTF-8</encoding>
            </configuration>
        </plugin>
    </plugins>
    
  </build>

</project>

The two references that I had to make were the following were one of jaxws-rt

<dependency>
  <groupId>com.sun.xml.ws</groupId>
  <artifactId>jaxws-rt</artifactId>
  <version>2.2</version>
</dependency>

and one for istack-commons-runtime.

<dependency>
  <groupId>com.sun.istack</groupId>
  <artifactId>istack-commons-runtime</artifactId>
  <version>2.22</version>
</dependency>

Service implementation

We now write our service implementation. For this purposes of this article will be very simple. I took over the pre-generated App.java and renamed it for my purposes to HelloService.java.

package org.test.ws;

import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebService;
import javax.jws.soap.SOAPBinding;

@WebService
@SOAPBinding(style = SOAPBinding.Style.RPC)
public class HelloService 
{
  @WebMethod(operationName = "sayHello")
  public String sayHello(@WebParam(name="guestname") String guestname) {
    if (guestname == null) { return "Hello"; }
    return "Hello " + guestname;
  }
}

Fairly basic, “hello” type service.

Endpoints

We instruct the jaxws framework that we have a service listening at any particular given endpoint by use of the sun-jaxws.xml file. Create this file in src/main/webapp/WEB-INF. It should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<endpoints xmlns="http://java.sun.com/xml/ns/jax-ws/ri/runtime" 
           version="2.0">
  <endpoint name="HelloService" 
            implementation="org.test.ws.HelloService" 
            url-pattern="/helloService" >
  </endpoint>
</endpoints>

To let Tomcat know from a deployment perspective what this application will handle, we also create a web.xml file that will be located in the same directory, src/main/webapp/WEB-INF. It looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" 
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         id="WebApp_ID" 
         version="2.5">
   
  <display-name>jaxwsExample</display-name>
 
  <listener>
    <listener-class>com.sun.xml.ws.transport.http.servlet.WSServletContextListener</listener-class>
  </listener>
  <servlet>
    <servlet-name>helloService</servlet-name>
    <servlet-class>com.sun.xml.ws.transport.http.servlet.WSServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>helloService</servlet-name>
    <url-pattern>/helloService</url-pattern>
  </servlet-mapping>
  <session-config>
    <session-timeout>120</session-timeout>
  </session-config>
</web-app>

Building and Running

At the console, you can now build this project:

mvn clean install

After you have deployed your war file to Tomcat, you service becomes available at http://localhost:8080/HelloService/helloService, this is if you’ve deployed locally; that is.

You’re offered a WSDL that your clients can use to integrate with your service http://localhost:8080/HelloService/helloService?WSDL.

Testing

Now that the service is up and running, we really want to test it to make sure it’s working. SOAP requests are HTTP POSTS. Sending the following request:

POST http://localhost:8080/HelloService/helloService HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: ""
Content-Length: 262
Host: 172.17.42.1:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
                  xmlns:ws="http://ws.test.org/">
  <soapenv:Header/>
    <soapenv:Body>
      <ws:sayHello>
        <guestname>Joe</guestname>
      </ws:sayHello>
    </soapenv:Body>
</soapenv:Envelope>

. . . should get you this response.

<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
  <S:Body>
    <ns2:sayHelloResponse xmlns:ns2="http://ws.test.org/">
      <return>Hello Joe</return>
    </ns2:sayHelloResponse>
  </S:Body>
</S:Envelope>