Data definition in Clojure

Introduction

In today’s article, I’ll briefly go over representing record-like information in Clojure.

Maps

First off, we’ll take a quick tour using a Map.

A Map is a collection that maps keys to values

So this is a pretty fundamental data type that we can use. We can pretty easily represent a person using it:

user=> (def person { :first-name "John" :last-name "Smith" :age 21 })
#'user/person
user=> user/person
{:first-name "John", :last-name "Smith", :age 21}

We can use count to count the number of pairs in the map.

user=> (count person)
3

We can use conj to give our person a hair attribute.

user=> (conj person { :hair :brown })
{:first-name "John", :last-name "Smith", :age 21, :hair :brown}

To navigate the map a little bit we can use get, contains?, find, keys, vals.

Using seq you can return a seq over the pairs; on each item in the seq you can use key to get the value of the key and val to get the value.

user=> (map #(println (key %1)) (seq person))
:first-name
:last-name
:age
(nil nil nil)

This gives us a full key/value pairing structure for us to arbitrarily represent data in a organized fashion; but not organized enough.

Records

Using records or defrecord we can turn a list of keys into a type that we can repeatably construct.

user=> (defrecord Person [ first-name last-name age ])
user.Person
user=> (def person-rec (->Person "Mary" "Jane" 25))
#'user/person-rec
user=> user/person-rec
#user.Person{:first-name "Mary", :last-name "Jane", :age 25}

We’re afforded all of the same functions above to work on this value; but we’re given the positional factory function ->Constructor so that we can construct our types in a much more intuitive way.

Domain functions

Now that we’ve spent a little bit of time creating data structures and defining data records, we can create functions that will allow us to perform operations on this data. Domain functions in Clojure perform operations specific to the data structure. In this following example, we’ve created a function to format a person’s full name:

user=> (defn get-full-name [p] (str (get p :first-name) " " (get p :last-name)))
#'user/get-name
user=> (user/get-full-name person)
"John Smith"

Object-oriented programming gives us polymorphism by allowing us to implement a class’s method differently per type that we derive from a common base. Clojure, gives us this multi-dispatch effect (choosing the right function for the given data-type) through multi methods and protocols.

Multi-methods

We’re going to use the defmulti macro to define our multi method and we’ll provide implementations using the defmethod macro. From the documentation:

Clojure eschews the traditional object-oriented approach of creating a new data type for each new situation, instead preferring to build a large library of functions on a small set of types. However, Clojure fully recognizes the value of runtime polymorphism in enabling flexible and extensible system architecture. Clojure supports sophisticated runtime polymorphism through a multimethod system that supports dispatching on types, values, attributes and metadata of, and relationships between, one or more arguments.

We’re going to expand the Person example above, by adding a new record of Company.

user=> (defrecord Person [first-name last-name age])
user=> (defrecord Company [name number])

Then we’ll define our multi method called get-full-name. Its job is to put together the name of our entity. Because we have both a Person and Company type entity, we’re going to need two different implementations:

user=> (defmulti get-full-name (fn [entity] (class entity)))
user=> (defmethod get-full-name Person [person] (str (get person :first-name) " " (get person :last-name)))
user=> (defmethod get-full-name Company [company] (get company :name))

The Person implementation of the get-full-name function concatenates the :first-name and :last-name attributes together, where as the Company implementation need only return the :name attribute.

Something that is interesting and unique to multi-methods is value-based dispatch; we’ve already seen type-based dispatch.

Consider a temperature conversion between Fahrenheit and Celsius. We create our multi method the same way, but this time we need to give the parameter values identity:

(defmulti convert-temp (fn [src-unit dest-unit amount] [src-unit dest-unit]))

We can now give our multi method some implementations based on the source and destination units passed:

;; C to F
(defmethod convert-temp [:c :f] [_ _ c] (+ 32 (* c 1.8)))

;; F to C
(defmethod convert-temp [:f :c] [_ _ f] (/ (- f 32) 1.8))

;; anything else
(defmethod convert-temp :default 
  [su du a] 
  (if (= su du) 
    a 
    (assert false "Conversion between units was not defined")))

We can now test out that the dispatching works:

user=> (convert-temp :f :c 100)
37.77777777777778
user=> (convert-temp :c :f 37.77777777777778)
100.0
user=> (convert-temp :c :c 50)
50
user=> (convert-temp :c :k 50)

AssertionError Assert failed: Conversion between units was not defined
false  user/eval1317/fn--1318 (form-init7827911538193486373.clj:1)

Protocols

Protocols are a little more reminiscent of object-oriented programming in a sense that they are closely related to interfaces. From the documentation:

Clojure is written in terms of abstractions. There are abstractions for sequences, collections, callability, etc. In addition, Clojure supplies many implementations of these abstractions. The abstractions are specified by host interfaces, and the implementations by host classes. While this was sufficient for bootstrapping the language, it left Clojure without similar abstraction and low-level implementation facilities. The protocols and datatypes features add powerful and flexible mechanisms for abstraction and data structure definition with no compromises vs the facilities of the host platform.

So, protocols give us a way to defining abstractions. We can treat our Person and Company scenario as such, by calling them a Party.

We use defprotocol to start our abstraction definition. extend-protocol is then used to supply implementations.

user=> (defprotocol Party
  #_=>   (get-full-name [entity]))

user=> (extend-protocol Party
  #_=>   Person
  #_=>   (get-full-name [person] (str (get person :first-name) " " (get person :last-name))))

user=> (extend-protocol Party
  #_=>   Company
  #_=>   (get-full-name [company] (get company :name)))

Wrapping up

This has been a brief tour on creating map/record data and some domain functions to work with them.

scipy

scipy is a specialist library for dealing with mathematical, science and engineering problems. According to their website:

The SciPy library, a collection of numerical algorithms and domain-specific toolboxes, including signal processing, optimization, statistics and much more.

The library itself is such a broad topic, so for the purposes of today’s article I just want to focus on a few items (from numpy) listed in their basic functions area covering polynomials and then finish up with the integration library.

Polynomials

A quick brush-up on some high school math (thanks wikipedia) tells us that a polynomial is an expression of more than two algebraic terms; these are normally variables (or indeterminates) and co-efficients.

Our polynomial with just one variable:

x² + 3x + 1

Let’s get scipy to represent this for us:

import numpy as np

p1 = np.poly1d([1, 3, 1])
print(p1)

This gives us an output of:

   2
1 x + 3 x + 1

Derivative

Lets derive this function, to find the formula that defines its instantaneous rate of change:

f(x)  = x² + 3x + 1
f'(x) = 2x + 3

We can get scipy to this for us using the deriv function:

print(p1.deriv())

Confirming everything for us, we now receive the derivative:

2 x + 3

Integral

We’ll integrate the same function now, providing us with the ability to calculate the area under the curve that the original polynomial would provide:

 f(x) = x² + 3x + 1
∫f(x) = x³ / 3 + (3x²) / 2 + x + C

We can simply use the integ function to do this for us, again:

print(p1.integ())

Providing us with the following answer:

        3       2
0.3333 x + 1.5 x + 1 x

General integration

Now that we’ve done some basic calculus with polynomials, we’ll get scipy to perform the integration for us. Using quad we specify the function that produces our values a maxima and minima value and then we’re given the integrated values.

We’ll simplify with x².

import numpy as np
import scipy.integrate as integrate

f = np.poly1d([1, 0, 0])
result = integrate.quad(f, 0, 1)

We’re then given the following:

(0.33333333333333337, 3.700743415417189e-15)

To my eye, the answer is ⅓, and performing the basic subtraction I’m given: 0.33333333333332965. Pretty close?

This has been a really light-on tour of Scipy!

requests

Getting your program to make simple HTTP has become a feature that is just a given these days. The python library requests makes life a lot easier.

Requests is the only Non-GMO HTTP library for Python, safe for human consumption.

Today’s post will focus on this library, to show you how you can be immediately productive.

How things were

One of the best comparisons comes directly from the requests documentation. Prior to its existence, you needed to do the following to perform a GET request with Basic Authentication:

import urllib2

gh_url = 'https://api.github.com'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

Bringing this code forward to use the requests library, the code simplifies:

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

Clearly, you can see that the code using the requests library is far simpler. Let’s look at some more examples.

Simple verbs

The simple verbs can be used quite simply by name:

requests.get('http://api.me.com/people/1')
requests.put('http://api.me.com/people/1', data = { 'name': 'john' })
requests.post('http://api.me.com/people', data = { 'name': 'john' })
requests.delete('http://api.me.com/people/1')

Of course the likes of OPTIONS and HEAD are also available.

Finishing up

The rest of this blog post could easily replicate all of the content in the documentation, but instead I’ll link to all of the good stuff:

• Quick start
• Advanced usage
• Authentication

numpy

numpy is an excellent library for dealing with array-based math problems. According to the website:

NumPy is the fundamental package for scientific computing with Python

In today’s post, I’ll go through some very basic operations of the library while piecing together the transformation segment that you might see in the back-end of a 3D rendering pipeline. We’ll focus on some vector and matrix math, but most importantly we’ll look at the creation of specialised matricies and matrix multiplication.

Vectors

All points in our 3D world is going to be re-presented by a 4D Homogenous co-ordinate. numpy provide the perfect abstraction here with the array function.

p = np.array([1, 0, 0, 1])

There are many other construction functions available for this single dimension value series that you can find in the documentation.

Once we’ve created the array, we can start to perform arithmetic on it.

>>> np.array([1, 0, 0, 1]) + np.array([0, 1, 0, 1])
array([1, 1, 0, 2])

>>> np.linalg.norm(np.array([1, 0, 0, 1]))
1.4142135623730951

>>> np.array([1, 0, 0, 1]).dot(np.array([0, 1, 0, 1]))
1

Matrices

Graduating from the vector, we now need a tabular representation for our linear algebraic operations that we’ll perform. To do this, we’ll use a matrix which numpy has a direct analog for.

First of all, every math library needs a shortcut to the identity matrix:

>>> np.identity(4)
array([[ 1.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.],
       [ 0.,  0.,  1.,  0.],
       [ 0.,  0.,  0.,  1.]])

Getting back into our 3D engine, we’re going to need a way to translate, scale and rotate the array points.

Translation

A translation matrix in 3-space using a 4x4 matrix looks like this:

[ 1 0 0 tx ]
[ 0 1 0 ty ]
[ 0 0 1 tz ]
[ 0 0 0 1  ]

So, when we apply this matrix to a column vector; we’ll end up with a simple addition of each of the components:

[ 1 0 0 tx ] [ vx ]   [ tx + vx ]
[ 0 1 0 ty ] [ vy ]   [ ty + vy ]
[ 0 0 1 tz ] [ vz ] = [ tz + vz ]
[ 0 0 0 1  ] [ 1  ]   [    1    ]

In python using numpy, this is greatly simplified:

def create_translation(x, y, z):
  return np.matrix([
      [1, 0, 0, x], 
      [0, 1, 0, y], 
      [0, 0, 1, z], 
      [0, 0, 0, 1]
  ])

With this function in hand, we can now start to move our vectors around in space:

# v sits at the origin
v = np.array([0, 0, 0, 1])

# translation matrix to move vectors along
# the x-axis by 1
t = create_translation(1, 0, 0)

# move v
t.dot(v)

With the output of this function:

matrix([[1, 0, 0, 1]])

Scale

We scale points geometry to adjust proportionality of objects. A scale matrix will do just this on any point around the origin 0, 0, 0, 1.

[ sx 0  0  0 ]
[ 0  sy 0  0 ]
[ 0  0  sz 0 ]
[ 0  0  0  1 ]

Applying a matrix of this shape to a column vector, we now get a multiplication effect:

[ sx 0  0  0 ] [ vx ]   [ sx * vx ]
[ 0  sy 0  0 ] [ vy ]   [ sy * vy ]
[ 0  0  sz 0 ] [ vz ] = [ sz * vz ]
[ 0  0  0  1 ] [ 1  ]   [    1    ]

The python for this process is pretty simple, again:

def create_scale(x, y, z):
  return np.matrix([
    [x, 0, 0, 0], 
    [0, y, 0, 0], 
    [0, 0, z, 0], 
    [0, 0, 0, 1]
  ])

Rotation

We’ll treat rotation on each axis separately.

x-axis rotation performs spherical movement between the y and z axis:

[  1   0     0    0  ]
[  0  cos∅ -sin∅  0  ]
[  0  sin∅  cos∅  0  ]
[  0   0     0    1  ]

y-axis rotation performs spherical movement between the x and z axis:

[  cos∅   0   sin∅  0 ]
[   0     1    0    0 ]
[ -sin∅   0   cos∅  0 ]
[   0     0    0    1 ]

z-axis rotation performs spherical movement between the x and y axis:

[  cos∅ -sin∅   0    0 ]
[  sin∅  cos∅   0    0 ]
[   0     0     1    0 ]
[   0     0     0    1 ]

These translate really well into python code, too!

def create_rotate_x(theta):
  ct = math.cos(theta)
  st = math.sin(theta)
  
  return np.matrix([
    [1, 0, 0, 0], 
    [0, ct, -st, 0], 
    [0, st, ct, 0], 
    [0, 0, 0, 1]
  ])

def create_rotate_y(theta):
  ct = math.cos(theta)
  st = math.sin(theta)
  
  return np.matrix([
    [ct, 0, st, 0], 
    [0, 1, 0, 0], 
    [-st, 0, ct, 0], 
    [0, 0, 0, 1]
  ])

def create_rotate_z(theta):
  ct = math.cos(theta)
  st = math.sin(theta)
  
  return np.matrix([
    [ct, -st, 0, 0], 
    [st, ct, 0, 0], 
    [0, 0, 1, 0], 
    [0, 0, 0, 1]
  ])

Armed with all of these functions, we have a basic (local) co-ordinate transform pipeline. We can start to make a little more sense out of these mathematical constructs by binding the results to variables that are named by their side effects.

# the cartesian plane moves positive, to the right
move_right_by_2 = create_translation(2, 0, 0)

# make any point twice the distance away from the origin
make_twice_bigger = create_scale(2, 2, 2)

# π radians is 180° 
turn_the_other_way = create_rotate_x(math.pi)

What is interesting about the matricies that we’ve just created, is that they’re reusable for as many points as we want. We can uniformly transform a group of vectors with the same matrix.

So, if I wanted to not only move right by 2, but also make twice bigger and turn the other way; I could multiply these matricies together to form a new transformation:

do_all_the_things = (turn_the_other_way * make_twice_bigger * move_right_by_2)

Now we can use do_all_the_things to transform our vector objects to perform all of these transformations at once:

>>> do_all_the_things.dot(np.array([0, 0, 1, 1]))
matrix([[  4.00000000e+00,  -2.44929360e-16,  -2.00000000e+00,
           1.00000000e+00]])

We did start, pointing towards the camera 0, 0, 1 and then after moving to the right by 2 and doubling our size, we’re now at 4 on the x-axis. We turn around about the x axis and we’re now facing away from the camera; twice the distance (-2 on the z-axis). Note that the y-axis has some epsilon garbage after our multiplies.

Well, it was a quick tour; but this has been numpy in (a little bit of) action.

Pyramid, Bottle and Tornado

As web application developers, we’re given a vast array of web application development frameworks at our disposal, In today’s post, I’m going to go through three of these; all based on the Python programming language. The frameworks are:

• Pyramid
• Bottle
• Tornado

These really are micro-frameworks for this purpose.

Pyramid

Pyramid, or the Pylons Project is a straight-forward application framework where most of the focus is placed on the application’s configuration. This isn’t an ancillary file supplied to the application, but defined in code, in module. From the web site:

Rather than focusing on a single web framework, the Pylons Project will develop a collection of related technologies. The first package from the Pylons Project was the Pyramid web framework. Other packages have been added to the collection over time, including higher-level components and applications. We hope to evolve the project into an ecosystem of well-tested, well-documented components which interoperate easily.

The Pylons project is a greater umbrella for the Pyramid-piece which is the web application framework.

Following is a “Hello, world” application using this framework.

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

def hello_world(request):
  return Response('Hello %(name)s!' % request.matchdict)

if __name__ == '__main__':
  config = Configurator()
  config.add_route('hello', '/hello/{name}')
  config.add_view(hello_world, route_name='hello')
  app = config.make_wsgi_app()
  server = make_server('0.0.0.0', 8080, app)
  server.serve_forever()

The Configurator class holding a lot of the application’s runtime, which is where routes and views come together.

Bottle

Bottle is a no-frills framework, with four main responsibilities: routing, templates, utilities and server.

It’s actually quite amazing (from a minimalist’s perspective) exactly how much you can get accomplished in such little code. Here’s the “Hello, world” example from their site:

from bottle import route, run, template

@route('/hello/<name>')
def index(name):
    return template('<b>Hello </b>!', name=name)

run(host='localhost', port=8080)

The simplistic feel to the framework certainly makes it very clear. template providing a direct text template with a model. run performing the job of the server and the @route attribute performing route configuration.

They’re faithful to their words:

Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library.

Tornado

Tornado is a web application framework that has been based around event-driven I/O. It’s going to be better suited to some of the persistent connection use-cases that some applications have (like long-polling or web sockets, etc). The following is from their site:

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user.

In its own way, Tornado can also be quite minimalist. Here’s their example:

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
  def get(self):
    self.write("Hello, world")

def make_app():
  return tornado.web.Application([
      (r"/", MainHandler),
  ])

if __name__ == "__main__":
  app = make_app()
  app.listen(8888)
  tornado.ioloop.IOLoop.current().start()

Key difference on this particular framework is the involvement of the IOLoop class. This really is event-driven web programming.