Starting a microservice with Scala

17 Feb 2017

A microservice is an architectural pattern that allows your services to deployed in an isolated fashion. This isolation allows your service to remain focused on its problem (and only its problem) that its trying to solve, as well as simplify telemetry, instrumentation, and measurement metrics. From Martin Fowler’s site:

The term “Microservice Architecture” has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.

If you want to learn more about microservices, seriously, check out google. They’re everywhere!

The purpose of today’s article is to stand a microservice up in Scala, to get up and running quickly.

Getting started

In a previous article, I showed you how you can create a scala project structure with a shell script. We’ll use that right now to create our project microservice-one.

$ new-scala-project microservice-one
$ cd microservice-one
$ tree
.
├── build.sbt
├── lib
├── project
├── src
│   ├── main
│   │   ├── java
│   │   ├── resources
│   │   └── scala
│   └── test
│       ├── java
│       ├── resources
│       └── scala
└── target

12 directories, 1 file

Dependencies

Now we’ll need to sort out our dependencies.

We’ll need scalatest for testing, akka and akka-http to help us make our API concurrent/parallel as well as available over HTTP. Our build.sbt file should look like this:

name := "microservice-one"
organization := "me.tuttlem"
version := "1.0.0"
scalaVersion := "2.12.1"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

libraryDependencies ++= {
  
  val akkaV       = "2.4.16"
  val akkaHttpV   = "10.0.1"
  val scalaTestV  = "3.0.1"
  
  Seq(
    "com.typesafe.akka" %% "akka-actor" % akkaV,
    "com.typesafe.akka" %% "akka-stream" % akkaV,
    "com.typesafe.akka" %% "akka-testkit" % akkaV,
    "com.typesafe.akka" %% "akka-http" % akkaHttpV,
    "com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpV,
    "com.typesafe.akka" %% "akka-http-testkit" % akkaHttpV,
    "org.scalatest"     %% "scalatest" % scalaTestV % "test"
  )
  
}

Update our project now:

$ sbt update

The code

We’re going to dump everything into one file today; the main application object. All of the parts are very descriptive though and I’ll go through each one. Our microservice is going to have one route, which is a GET on /greeting. It’ll return us a simple message.

First up, we model how the message will look:

case class Greeting(message: String)

Using this case class, you’d expect messages to be returned that look like this:

{ message: "Here is the message!" }

We tell the application how to serialize this data over the http channel using Protocols:

trait Protocols extends DefaultJsonProtocol {
  implicit val greetingFormat = jsonFormat1(Greeting.apply)
}

Now, we can put together our actual service implementation. Take a look specifically at the DSL that scala is provided for route definition:

trait Service extends Protocols {
  implicit val system: ActorSystem
  implicit def executor: ExecutionContextExecutor
  implicit val materializer: Materializer

  def config: Config
  val logger: LoggingAdapter

  val routes = {
    logRequestResult("microservice-one") {
      pathPrefix("greeting") {
      	get {
      		complete(Greeting("Hello to you!"))
      	}
      }
    }
  }

}

So, our one route here will constantly just send out “Hello to you!”.

Finally, all of this gets hosted in our main application object:

object MicroserviceOne extends App with Service {
  override implicit val system = ActorSystem()
  override implicit val executor = system.dispatcher
  override implicit val materializer = ActorMaterializer()

  override val config = ConfigFactory.load()
  override val logger = Logging(system, getClass)

  Http().bindAndHandle(routes, config.getString("http.interface"), config.getInt("http.port"))
}

That’s it for the code. In the src/main/resources directory, we’ll put a application.conf file that details a few configurations for us:

akka {
	loglevel = DEBUG
}

http {
	interface = "0.0.0.0"
	port = 3000
}

Running

Lets give it a run now.

$ sbt run

Once SBT has finished its dirty work, you’ll be able to request your route at http://localhost:3000/greeting:

$ curl -v http://localhost:3000/greeting
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET /greeting HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: akka-http/10.0.1
< Date: Thu, 16 Feb 2017 22:37:52 GMT
< Content-Type: application/json
< Content-Length: 27
< 
* Connection #0 to host localhost left intact
{"message":"Hello to you!"}

Perfect.

That’s all for today.

Create a UDF for Hive with Scala

17 Feb 2017

In today’s post, I’m going to walk through the basic process of creating a user defined function for Apache Hive using the Scala.

A quick _but important_ note: I needed to use the JDK 1.7 to complete the following. Using 1.8 saw errors that suggested that Hive on my distribution of Hadoop was not supported.

Setup your project

Create an sbt-based project, and start off adding the following to your project/assembly.sbt.

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

What this had added is the sbt-assembly to your project. This allows you to bundle your scala application up as a fat JAR. When we issue the command sbt assemble at the console, we invoke this plugin to construct the fat JAR for us.

Now we fill out the build.sbt. We need to reference an external JAR, called hive-exec. This JAR is available by itself from the maven repository. I took a copy of mine from the hive distribution installed on my server. Anyway, it lands in the project’s lib folder.

name := "hive-udf"
version := "1.0"
scalaVersion := "2.11.1"
unmanagedJars in Compile += file("./lib/hive-exec-2.1.1.jar")

Write your function

Now it’s time to actually start writing some functions. In the following module, we’re just performing some basic string manipulation with trim, toUpperCase and toLowerCase. Each of which is contained in its own class, deriving from the UDF type:

scala/StringFunctions.scala

package me.tuttlem.udf

import org.apache.hadoop.hive.ql.exec.UDF

class TrimString extends UDF {
  def evaluate(str: String): String = {
    str.trim
  }
}

class UpperCaseString extends UDF {
  def evaluate(str: String): String = {
    str.toUpperCase
  }
}

class LowerCaseString extends UDF {
  def evaluate(str: String): String = {
    str.toLowerCase
  }
}

Now that we’ve written all of the code, it’s time to compile and assemble our JAR:

$ sbt assemble

To invoke

Copying across the JAR into an accessible place for hive is the first step here. Once that’s done, we can start up the hive shell and add it to the session:

ADD JAR /path/to/the/jar/my-udfs.jar;

Then, using the CREATE FUNCTION syntax, we can start to reference pieces of our module:

CREATE FUNCTION trim as 'me.tuttlem.udf.TrimString';
CREATE FUNCTION toUpperCase as 'me.tuttlem.udf.UpperCaseString';
CREATE FUNCTION toLowerCase as 'me.tuttlem.udf.LowerCaseString';

We can now use our functions:

hive> CREATE FUNCTION toUpperCase as 'me.tuttlem.udf.UpperCaseString';
OK
Time taken: 0.537 seconds
hive> SELECT toUpperCase('a test string');
OK
A TEST STRING
Time taken: 1.399 seconds, Fetched: 1 row(s)

hive> CREATE FUNCTION toLowerCase as 'me.tuttlem.udf.LowerCaseString';
OK
Time taken: 0.028 seconds
hive> SELECT toLowerCase('DON\'T YELL AT ME!!!');
OK
don't yell at me!!!
Time taken: 0.093 seconds, Fetched: 1 row(s)

That’s it!

Creating a Scala SBT project structure

16 Feb 2017

Today’s post is going to be a tip on creating a project structure for your Scala projects that is SBT ready. There’s no real magic to it, just a specific structure that you can easily bundle up into a console application.

The shell script

To kick start your project, you can simple use the following shell script:

#!/bin/zsh
mkdir $1
cd $1

mkdir -p src/{main,test}/{java,resources,scala}
mkdir lib project target

echo 'name := "$1"
version := "1.0"
scalaVersion := "2.10.0"' > build.sbt

cd ..

This will give you everything that you need to get up an running. You’ll now have a structure like the following to work with:

.
├── build.sbt
├── lib
├── project
├── src
│   ├── main
│   │   ├── java
│   │   ├── resources
│   │   └── scala
│   └── test
│       ├── java
│       ├── resources
│       └── scala
└── target

Spellcheck for Sublime

13 Feb 2017

Today’s post is a very quick tutorial on turning on spell check, for Sublime Text.

Looking at the documentation you can add the following:

"spell_check": true,
"dictionary": "Packages/Language - English/en_US.dic"

Easy.

Managing multiple SSH identities

13 Feb 2017

Sometimes, it makes sense to have multiple SSH identites. This can certainly be the case if you’re doing work with your own personal accounts, vs. doing work for your job. You’re not going to want to use your work account for your personal stuff.

In today’s post, I’m going to run through the few steps that you need to take in order to manage multiple SSH identities.

Different identities

First up, we generate two different identities:

ssh-keygen -t rsa -C "user@work.com"

When asked, make sure you give the file a unique name:

Enter file in which to save the key (/home/michael/.ssh/id_rsa): ~/.ssh/id_rsa_work

Now, we create the identity for home.

ssh-keygen -t rsa -C "user@home.com"

Again, set the file name so they don’t collide:

Enter file in which to save the key (/home/michael/.ssh/id_rsa): ~/.ssh/id_rsa_home

Now, we should have the following:

id_rsa_home
id_rsa_home.pub
id_rsa_work
id_rsa_work.pub

Configuration

Now we create a configuration file that ties all of the identities up. Start editing ~/.ssh/config:

# Home account
Host home-server.com
  HostName home-server.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_rsa_home

# Company account
Host work-server.com
  HostName work-server.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_rsa_work

Delete all of the cached keys:

ssh-add -D

If you see the error message Could not open a connection to your authentication agent. you’ll need to run the following:

ssh-agent -s

Add your keys

You can now list your keys with:

ssh-add -l

You can add your keys back in with the following:

ssh-add ~/.ssh/id_rsa_work
ssh-add ~/.ssh/id_rsa_home