Create a DSL using implicits in Scala

09 Jan 2017

Scala provides some simple options to get up and running the development of domain specific languages. In today’s post, I’ll take you through the usage of the implicit keyword that easily allows for this to happen.

End goal

For the purposes of today, I want to create a language that defines simple, directional instructions. “Move left”, “move forward”, etc. The end goal, I want a language that looks like this:

var jump = 5.0 units up
var shuffle = -2.0 units left
var fall = -6000.0 units down

Implicit conversion

The implicit conversion will allow us to get this process started. It’ll provide the ability for us to take that literal, double value (at the start of each of those statements), and convert them into a class instance that we can do more interesting things with.

implicit def convertDoubleToDirectionUtil(amount: Double) = new DirectionUtil(amount)

Ok, so we don’t know what a DirectionUtil class is just yet. That’s coming. Right now, it’s important to focus on this function which is an implicit conversion; allows us to express a double-value like normal and have Scala treat it as a DirectionUtil instance.

Implicit classes

The implicit class allows us to offer a class up to an implicit conversion when it’s in scope.

The definition of our DirectionUtil class here, allows us the units definition function.

implicit class DirectionUtil(val amount: Double) {

  def units(dir: String) = {
    dir match {
      case "forward" => scale((0.0, 0.0, 1.0), amount)
      case "backward" => scale((0.0, 0.0, -1.0), amount)
      case "left" => scale((-1.0, 0.0, 0.0), amount)
      case "right" => scale((1.0, 0.0, 0.0), amount)
      case "up" => scale((0.0, -1.0, 0.0), amount)
      case "down" => scale((0.0, 1.0, 0.0), amount)
      case _ => (0.0, 0.0, 0.0)
    }
  }

}

Really simply, this is just a vector-scaling function. Takes in a tuple, and adjusts it by a magnitude depending on the direction passed in.

Wrapping it all up

To finish up, we’ll put this class and implicit conversion into one neat singleton:

object DirectionUtil {
  val forward = "forward"
  val backward = "backward"
  val left = "left"
  val right = "right"
  val up = "up"
  val down = "down"

  def scale(vec: (Double, Double, Double), mag: Double) = {
    val (x, y, z) = vec
    (x * mag, y * mag, z * mag)
  }

  implicit def convertDoubleToDirectionUtil(amount: Double) = new DirectionUtil(amount)

  implicit class DirectionUtil(val amount: Double) {

    def units(dir: String) = {
      dir match {
        case "forward" => scale((0.0, 0.0, 1.0), amount)
        case "backward" => scale((0.0, 0.0, -1.0), amount)
        case "left" => scale((-1.0, 0.0, 0.0), amount)
        case "right" => scale((1.0, 0.0, 0.0), amount)
        case "up" => scale((0.0, -1.0, 0.0), amount)
        case "down" => scale((0.0, 1.0, 0.0), amount)
        case _ => (0.0, 0.0, 0.0)
      }
    }

  }
}

Now that we have this defined, we can start writing code that looks like this:

object Main extends App {
  import DirectionUtil._

  val jump = 5.0 units up
  println(jump)
}

. . which, as expected gives us an output like this:

(0.0,-5.0,0.0)

In closing

This is a really simple example, but you can see immediately how it can be applied to much more complex scenarios; and how you can be a lot more expressive in your scala source code.

Lists, vectors, sets and maps

15 Dec 2016

In a previous article, we’d spent a little bit of time working with maps in clojure; we also looked at formalising the structure of a map into records. In today’s post, I’ll discuss some of the collection types that will allows us to carry around many values at once.

General

Before we start looking at the collection types, there are some general-purpose functions that you can use with these types. These are very helpful functions that really form the basis of your collection processing:

first will return you the head of your collection; with rest providing you with everything else but the head. last will return you the final item in your collection and cons will provide you with the ability to construct a new list with a given head and tail.

nth will give you indexed access to you collection.

List

A list is an ordered collection of elements. From the documentation:

Lists are collections. They implement the ISeq interface directly (except for the empty list, which is not a valid seq). count is O(1). conj puts the item at the front of the list.

Because lists are a natural part of the language, they’ll be invoked as a function if they’re not defined correctly or quoted adequately. A list can be defined using the list function, but can also be quoted:

user=> (list :gibson :fender :ibanez)
(:gibson :fender :ibanez)
user=> '(:joe :john :peter)
(:joe :john :peter)
user=> '("a" "b" "c"))
("a" "b" "c")
user=> '(1 2 3)
(1 2 3)

Vector

A vector is an ordered collection of elements, but is optimised for the random-access case. From the documentation:

A Vector is a collection of values indexed by contiguous integers. Vectors support access to items by index in log32N hops. count is O(1). conj puts the item at the end of the vector.

user=> [:gibson :fender :ibanez]
[:gibson :fender :ibanez]

You can use the nth function to get the value at an index, or you can just invoke the vector itself. It’ll take an index, and return you the item:

user=> (nth [:gibson :fender :ibanez] 1)
:fender
user=> ([:gibson :fender :ibanez] 1)
:fender

Set

A set is an un-ordered collection of items. You can use sorted-set if your application requires your set to remain in an ordered way.

Clojure allows you to wrap your values with #{} in order to define a set, literally.

user=> #{:gibson :fender ibanez}
#{:gibson :fender ibanez}

Sets have methods to easily union two sets, or find the difference.

Map

Definition

A map is a collection of key-value-pairs. Declaring maps allows you to define attribute keys and values in an organised or arbitrary way. From the documentation:

A Map is a collection that maps keys to values. Two different map types are provided - hashed and sorted. Hash maps require keys that correctly support hashCode and equals. Sorted maps require keys that implement Comparable, or an instance of Comparator. Hash maps provide faster access (log32N hops) vs (logN hops), but sorted maps are, well, sorted. count is O(1). conj expects another (possibly single entry) map as the item, and returns a new map which is the old map plus the entries from the new, which may overwrite entries of the old.

user=> { :name "Michael" :age 21 }
{:name "Michael", :age 21}
user=> ({ :name "Michael" :age 21 } :name)
"Michael"

user=> (def person { :name "Michael" :age 21 })
#'user/person
user=> (assoc person :age 22)
{:name "Michael", :age 22}

user=> (def person-aux { :hair :brown })
#'user/person-aux
user=> (merge person person-aux)
{:name "Michael", :age 21, :hair :brown}

Finishing up

This has been a quick lap around the collection types in Clojure.

Data definition in Clojure

07 Dec 2016

Introduction

In today’s article, I’ll briefly go over representing record-like information in Clojure.

Maps

First off, we’ll take a quick tour using a Map.

A Map is a collection that maps keys to values

So this is a pretty fundamental data type that we can use. We can pretty easily represent a person using it:

user=> (def person { :first-name "John" :last-name "Smith" :age 21 })
#'user/person
user=> user/person
{:first-name "John", :last-name "Smith", :age 21}

We can use count to count the number of pairs in the map.

user=> (count person)
3

We can use conj to give our person a hair attribute.

user=> (conj person { :hair :brown })
{:first-name "John", :last-name "Smith", :age 21, :hair :brown}

To navigate the map a little bit we can use get, contains?, find, keys, vals.

Using seq you can return a seq over the pairs; on each item in the seq you can use key to get the value of the key and val to get the value.

user=> (map #(println (key %1)) (seq person))
:first-name
:last-name
:age
(nil nil nil)

This gives us a full key/value pairing structure for us to arbitrarily represent data in a organized fashion; but not organized enough.

Records

Using records or defrecord we can turn a list of keys into a type that we can repeatably construct.

user=> (defrecord Person [ first-name last-name age ])
user.Person
user=> (def person-rec (->Person "Mary" "Jane" 25))
#'user/person-rec
user=> user/person-rec
#user.Person{:first-name "Mary", :last-name "Jane", :age 25}

We’re afforded all of the same functions above to work on this value; but we’re given the positional factory function ->Constructor so that we can construct our types in a much more intuitive way.

Domain functions

Now that we’ve spent a little bit of time creating data structures and defining data records, we can create functions that will allow us to perform operations on this data. Domain functions in Clojure perform operations specific to the data structure. In this following example, we’ve created a function to format a person’s full name:

user=> (defn get-full-name [p] (str (get p :first-name) " " (get p :last-name)))
#'user/get-name
user=> (user/get-full-name person)
"John Smith"

Object-oriented programming gives us polymorphism by allowing us to implement a class’s method differently per type that we derive from a common base. Clojure, gives us this multi-dispatch effect (choosing the right function for the given data-type) through multi methods and protocols.

Multi-methods

We’re going to use the defmulti macro to define our multi method and we’ll provide implementations using the defmethod macro. From the documentation:

Clojure eschews the traditional object-oriented approach of creating a new data type for each new situation, instead preferring to build a large library of functions on a small set of types. However, Clojure fully recognizes the value of runtime polymorphism in enabling flexible and extensible system architecture. Clojure supports sophisticated runtime polymorphism through a multimethod system that supports dispatching on types, values, attributes and metadata of, and relationships between, one or more arguments.

We’re going to expand the Person example above, by adding a new record of Company.

user=> (defrecord Person [first-name last-name age])
user=> (defrecord Company [name number])

Then we’ll define our multi method called get-full-name. Its job is to put together the name of our entity. Because we have both a Person and Company type entity, we’re going to need two different implementations:

user=> (defmulti get-full-name (fn [entity] (class entity)))
user=> (defmethod get-full-name Person [person] (str (get person :first-name) " " (get person :last-name)))
user=> (defmethod get-full-name Company [company] (get company :name))

The Person implementation of the get-full-name function concatenates the :first-name and :last-name attributes together, where as the Company implementation need only return the :name attribute.

Something that is interesting and unique to multi-methods is value-based dispatch; we’ve already seen type-based dispatch.

Consider a temperature conversion between Fahrenheit and Celsius. We create our multi method the same way, but this time we need to give the parameter values identity:

(defmulti convert-temp (fn [src-unit dest-unit amount] [src-unit dest-unit]))

We can now give our multi method some implementations based on the source and destination units passed:

;; C to F
(defmethod convert-temp [:c :f] [_ _ c] (+ 32 (* c 1.8)))

;; F to C
(defmethod convert-temp [:f :c] [_ _ f] (/ (- f 32) 1.8))

;; anything else
(defmethod convert-temp :default 
  [su du a] 
  (if (= su du) 
    a 
    (assert false "Conversion between units was not defined")))

We can now test out that the dispatching works:

user=> (convert-temp :f :c 100)
37.77777777777778
user=> (convert-temp :c :f 37.77777777777778)
100.0
user=> (convert-temp :c :c 50)
50
user=> (convert-temp :c :k 50)

AssertionError Assert failed: Conversion between units was not defined
false  user/eval1317/fn--1318 (form-init7827911538193486373.clj:1)

Protocols

Protocols are a little more reminiscent of object-oriented programming in a sense that they are closely related to interfaces. From the documentation:

Clojure is written in terms of abstractions. There are abstractions for sequences, collections, callability, etc. In addition, Clojure supplies many implementations of these abstractions. The abstractions are specified by host interfaces, and the implementations by host classes. While this was sufficient for bootstrapping the language, it left Clojure without similar abstraction and low-level implementation facilities. The protocols and datatypes features add powerful and flexible mechanisms for abstraction and data structure definition with no compromises vs the facilities of the host platform.

So, protocols give us a way to defining abstractions. We can treat our Person and Company scenario as such, by calling them a Party.

We use defprotocol to start our abstraction definition. extend-protocol is then used to supply implementations.

user=> (defprotocol Party
  #_=>   (get-full-name [entity]))

user=> (extend-protocol Party
  #_=>   Person
  #_=>   (get-full-name [person] (str (get person :first-name) " " (get person :last-name))))

user=> (extend-protocol Party
  #_=>   Company
  #_=>   (get-full-name [company] (get company :name)))

Wrapping up

This has been a brief tour on creating map/record data and some domain functions to work with them.

scipy

01 Dec 2016

scipy is a specialist library for dealing with mathematical, science and engineering problems. According to their website:

The SciPy library, a collection of numerical algorithms and domain-specific toolboxes, including signal processing, optimization, statistics and much more.

The library itself is such a broad topic, so for the purposes of today’s article I just want to focus on a few items (from numpy) listed in their basic functions area covering polynomials and then finish up with the integration library.

Polynomials

A quick brush-up on some high school math (thanks wikipedia) tells us that a polynomial is an expression of more than two algebraic terms; these are normally variables (or indeterminates) and co-efficients.

Our polynomial with just one variable:

x² + 3x + 1

Let’s get scipy to represent this for us:

import numpy as np

p1 = np.poly1d([1, 3, 1])
print(p1)

This gives us an output of:

   2
1 x + 3 x + 1

Derivative

Lets derive this function, to find the formula that defines its instantaneous rate of change:

f(x)  = x² + 3x + 1
f'(x) = 2x + 3

We can get scipy to this for us using the deriv function:

print(p1.deriv())

Confirming everything for us, we now receive the derivative:

2 x + 3

Integral

We’ll integrate the same function now, providing us with the ability to calculate the area under the curve that the original polynomial would provide:

 f(x) = x² + 3x + 1
∫f(x) = x³ / 3 + (3x²) / 2 + x + C

We can simply use the integ function to do this for us, again:

print(p1.integ())

Providing us with the following answer:

        3       2
0.3333 x + 1.5 x + 1 x

General integration

Now that we’ve done some basic calculus with polynomials, we’ll get scipy to perform the integration for us. Using quad we specify the function that produces our values a maxima and minima value and then we’re given the integrated values.

We’ll simplify with x².

import numpy as np
import scipy.integrate as integrate

f = np.poly1d([1, 0, 0])
result = integrate.quad(f, 0, 1)

We’re then given the following:

(0.33333333333333337, 3.700743415417189e-15)

To my eye, the answer is ⅓, and performing the basic subtraction I’m given: 0.33333333333332965. Pretty close?

This has been a really light-on tour of Scipy!

requests

01 Dec 2016

Getting your program to make simple HTTP has become a feature that is just a given these days. The python library requests makes life a lot easier.

Requests is the only Non-GMO HTTP library for Python, safe for human consumption.

Today’s post will focus on this library, to show you how you can be immediately productive.

How things were

One of the best comparisons comes directly from the requests documentation. Prior to its existence, you needed to do the following to perform a GET request with Basic Authentication:

import urllib2

gh_url = 'https://api.github.com'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

Bringing this code forward to use the requests library, the code simplifies:

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

Clearly, you can see that the code using the requests library is far simpler. Let’s look at some more examples.

Simple verbs

The simple verbs can be used quite simply by name:

requests.get('http://api.me.com/people/1')
requests.put('http://api.me.com/people/1', data = { 'name': 'john' })
requests.post('http://api.me.com/people', data = { 'name': 'john' })
requests.delete('http://api.me.com/people/1')

Of course the likes of OPTIONS and HEAD are also available.

Finishing up

The rest of this blog post could easily replicate all of the content in the documentation, but instead I’ll link to all of the good stuff:

Older Newer

Cogs and Levers A blog full of technical stuff

Create a DSL using implicits in Scala

End goal

Implicit conversion

Implicit classes

Wrapping it all up

In closing

Lists, vectors, sets and maps

General

List

Vector

Set

Map

Definition

Finishing up

Data definition in Clojure

Introduction

Maps

Records

Domain functions

Multi-methods

Protocols

Wrapping up

scipy

Polynomials

Derivative

Integral

General integration

requests

How things were

Simple verbs

Finishing up