Hadoop 2 (2.2.0) setup on Debian

09 Jan 2014

Today’s post will just be a walk through of the steps required to install Hadoop 2 on Debian Linux. Please note that this is for a single node installation only. This guide is heavily based on the Ubuntu instructions found here.

Install Java

	
# install the java jdk
$ sudo apt-get install openjdk-7-jdk
 
# make a jdk symlink
$ cd /usr/lib/jvm
$ ln -s java-7-openjdk-amd64 jdk
 
# make sure that ssh server is installed
$ sudo apt-get install openssh-server

Add Hadoop Users and Groups

	
# create a new group for hadoop
$ sudo addgroup hadoop
 
# create the hduser and put them in the hadoop group
$ sudo adduser --ingroup hadoop hduser
 
# add them to the sudo group also
$ sudo adduser hduser sudo

Now login as “hduser”.

SSH Certificates

	
# generate your key
$ ssh-keygen -t rsa -P ''
 
# set your public key as authorized
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
 
# test out ssh
$ ssh localhost

Download Hadoop

	
# downoad the package
$ cd ~
$ wget http://mirror.rackcentral.com.au/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
 
# extract the package
$ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local
$ cd /usr/local
$ sudo mv hadoop-2.2.0 hadoop
 
# get the hduser to take ownership
$ sudo chown -R hduser:hadoop hadoop

Setup Environment Variables

Add the following lines to your ~/.bashrc

# Hadoop variables
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL

Add the following lines to /usr/local/hadoop/etc/hadoop/hadoop-env.sh

# modify JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/jdk/

Re-login to your machine as hduser, and check the hadoop version.

$ hadoop version

Configure Hadoop

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/core-site.xml

<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/yarn-site.xml

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Make a copy of the mapred-site template file

	
$ mv mapred-site.xml.template mapred-site.xml
$ vi mapred-site.xml

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/mapred-site.xml

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

Prepare the Filesystem

	
# create the physical directories
$ cd ~
$ mkdir -p mydata/hdfs/namenode
$ mkdir -p mydata/hdfs/datanode

Add the following lines into the <configuration> node /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/mydata/hdfs/datanode</value>
 </property>

Format the namenode

	
$ hdfs namenode -format

Start Hadoop

	
$ start-dfs.sh
$ start-yarn.sh
 
# check that services are running
$ jps

Run the Example

	
$ cd /usr/local/hadoop
$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

Getting started with Lua using C++

08 Jan 2014

Introduction

Lua is a programming language that has seen increased popularity from the game development industry. It’s put to use in the disciplines of providing configuration data all the way through to scripting automated character actions.

In today’s post, I’ll walk you through the setup process of the libraries up to writing your first testing application within a Linux environment.

Library Installation

Before you get downloading and building this library, you’ll just need to ensure that you have a build environment installed and are able to compile.

At the time of writing this article, the Lua website had 5.2.3 as their latest release. From their downloads page, grab the latest tar.gz archive and extract it onto your machine. Following along with their building instructions, issuing “make linux test” whilst in the extracted folder did the trick. It built Lua ready for me to use. A nice convenience of the make file was the “make local” option. Upon issuing this request, the make system will prepare an installation folder that is suitable for you to use locally (i.e. not installed to the system).

When it comes to downloading the latest versions of libraries, I’ll rarely install these to my system. Rather, I drag them around for each project that needs them so that the project determines its dependencies as opposed to my system.

From here, I prepare a distributable directory of development files that I know that each of my project needs. In the case of Lua, I create the following structure:

.
└── lua-5.2.3
    ├── include
    │   ├── lauxlib.h
    │   ├── luaconf.h
    │   ├── lua.h
    │   ├── lua.hpp
    │	  └── lualib.h
    └── lib
    	├── liblua.a
    	└── lua
    		└── 5.2

I have all of the development headers available (under the “include” folder) and a static version of the Lua library (under lib).

Building applications

When building Lua applications, you’ll need to specify the libraries and include folders to your compiler so it knows where to find them. For a test application that I’d written, the following command line compiled an application for me without any trouble.

$ g++ -Ilib/lua-5.2.3/include -Llib/lua-5.2.3/lib/ -llua -ldl

You can see at the end there, mention of both the “lua” and “dl” libraries.

Test application

A very simple test to start will be creating a program that will execute a Lua script, pull out on of its global variables and display it to screen.

Here’s our Lua script:

x = 10

Pretty simple. We have one variable x set to 10. Now here’s the C++ code that we use to read that one variable out and present it to screen.

#include <iostream>
#include <lua.hpp>

int main(int argc, char *argv[]) {
  // create a new lua context to work with
  lua_State *L = luaL_newstate();

  // open any library we may use
  luaL_openlibs(L);

  // read the Lua script off disk and execute it
  if ((luaL_dofile(L, "test.lua")) != 0) {
    
    // handle any errors 
    std::cout << "unable to load test.lua" << std::endl;
    return 1;
    
  }

  // put the value of X at the top of the stack
  lua_getglobal(L, "x");
  
  // interpret the value at the top of the stack 
  // as an integer and put it in the variable "val"
  int val = (int)lua_tointeger(L, -1);
  
  // pop the value of X off the stack
  lua_pop(L, 1);

  // write the value out
  std::cout << "Value of X: " << val << std::endl;

  // finish up with the Lua context
  lua_close(L);

  return 0;

}

I think that the source code above (paired with the documentation on the Lua website) should make things pretty straight forward.

That’s it for today. This is only scratching the surface on what Lua can do. For my purposes right now, I just need to feed configuration values into programs, this fits the purpose really well.

Installing SDL2 on Linux

05 Jan 2014

Introduction

SDL2 was released a little while ago, but still hasn’t made it into the stable repositories of some Linux distributions. After doing a big of digging, it’s not too hard to get this installed yourself - most of the advice offered in this post comes from an answer on the Ubuntu forums here.

In today’s post, we’ll install SDL2 on a Debian/Ubuntu style distribution from source.

Dependencies

First thing before we download and compile the SDL2 source is to get some of the dependencies installed on your system. The following install line will put all of the libraries that SDL2 requires:

$ sudo apt-get install build-essential xorg-dev libudev-dev libts-dev libgl1-mesa-dev libglu1-mesa-dev libasound2-dev libpulse-dev libopenal-dev libogg-dev libvorbis-dev libaudiofile-dev libpng12-dev libfreetype6-dev libusb-dev libdbus-1-dev zlib1g-dev libdirectfb-dev

Once all of these have installed successfully, you’ll need to download a copy of the source. All downloads can be found here. This guide will assume that you’ll download the .tar.gz source archive.

Compilation and Installation

Extract the package into a directory under your home directory, somewhere . . .

	
$ tar -xzf SDL2-*.tar.gz
$ cd SDL2-*

Next we’ll configure, build and install the libraries.

	
$ ./configure
$ make
$ sudo make install

Once compilation and installation have complete, you’ll need to update your library links/cache. Do this with the following command:

	
$ sudo ldconfig

That’s all there is to it.

Perlin Noise

31 Dec 2013

Introduction

Noise functions in computer applications allow programmers to make the machine act a little more naturally. It’s the randomness introduced with these algorithms that gives the computer what appears to be “free thought” or unexpected decisions.

Today, I’ll walk through the Perlin Noise algorithm which has applications in computer science ranging from player movement, landscape generation, clouds, etc.

Here are some examples of the Perlin Noise function output into two dimensions:

Noise 1 Noise 2

In today’s post, I’ll walk through the Perlin Noise algorithm and what steps you need to take to implement it yourself.

Understanding Noise

The Perlin Noise algorithm can be broken down into a few smaller pieces to make it easier to understand. At its heart, the algorithm needs pseudo-random numbers. These random numbers should be repeatable so that you can re-generate the same noise pattern at will.

A common noise function for two parameters that I have found used over the web is as follows:

float noise(const int x, const int y) {
  int n = x + y * 57;
  n = (n << 13) ^ n;
  return (1.0f - ( ( n * (n * n * 15731 + 789221) + 1376312589) & 0x7FFFFFFF) / 1073741824.0f);
}

There’s a lot of math transformation in this previous function. You can use any function at all to produce your random numbers, just make sure that you can generate them against two parameters (in the case of 2d) and that you’ll get repeatable results.

Next we’ll smooth out the noise between two points. We’ll do this by sampling the corners, sides and centre of the point we’re currently generating for.

float smoothNoise(const float x, const float y) {
  int ix = (int)x;
  int iy = (int)y;

  // sample the corners
  float corners = (noise(ix - 1, iy - 1) +
                   noise(ix + 1, iy - 1) +
                   noise(ix - 1, iy + 1) +
                   noise(ix + 1, iy + 1)) / 16;

  // sample the sides
  float sides = (noise(ix - 1, iy) +
                 noise(ix + 1, iy) +
                 noise(ix, iy - 1) +
                 noise(ix, iy + 1)) / 8;

  // sample the centre
  float centre = noise(ix, iy);

  // send out the accumulated result
  return corners + sides + centre;
}

With the above function, we can now sample a small area for a given point. All based on our random number generator.

For the fractional parts that occur between solid boundaries, we’ll use a specific interpolation method. I’ve defined two below. One that will do linear interpolation and one that will use cosine for a smoother transition between points.

/* Linear interpolation */
float lerp(float a, float b, float x) {
   return a * (1 - x) + b * x;
}

/* Trigonometric interpolation */
float terp(float a, float b, float x) {
  float ft = x * 3.1415927f;
  float f = (1 - cosf(ft)) * 0.5f;

  return a * (1 - f) + b * f;
}

/* Noise interpolation */
float interpolateNoise(const float x, const float y) {
  int   ix = (int)x;
  float fx = x - ix;

  int   iy = (int)y;
  float fy = y - iy;

  float v1 = smoothNoise(ix, iy),
        v2 = smoothNoise(ix + 1, iy),
        v3 = smoothNoise(ix, iy + 1),
        v4 = smoothNoise(ix + 1, iy + 1);

  float i1 = terp(v1, v2, fx),
        i2 = terp(v3, v4, fx);

  return terp(i1, i2, fy);
}

Finally we use this smooth interpolation to perform the perlin noise function. A couple of interesting co-effecients that are provided to the algorithm are “octaves” and “persistence”. “octaves” defines how many iterations that will be performed and “persistence” defines how much of the spectrum we’ll utilise. It’s highly interactive to the main curve co-effecients: frequency and amplitude.

float perlin2d(const float x, const float y,
               const int octaves, const float persistence) {
  float total = 0.0f;

  for (int i = 0; i <= (octaves - 1); i ++) {
    float frequency = powf(2, i);
    float amplitude = powf(persistence, i);

    total = total + interpolateNoise(x * frequency, y * frequency) * amplitude;
  }

  return total;
}

This now provides us with a way to create a map (or 2d array) to produce images much like I’d pasted in above. Here is a typical build loop.

for (int x = 0; x < w; x ++) {
  for (int y = 0; y < h; y ++) {
    float xx = (float)x / (float)this->width;
    float yy = (float)y / (float)this->height;

    map[x + (y * w)] = perlin::perlin2d(
      xx, yy,
      6, 1.02f
    );
  }
}

That should be enough to get you going.

Follow up to a Camera Implementation

30 Dec 2013

Introduction

In a previous post I’d written about a simple camera implementation that you can use in projects of your own. This post I’ll show how I’ve implemented this camera with some mouse handling routines to make it feel like you’re orienting your head with the mouse.

The Idea

We’ll capture all of the mouse movements given in an application and see how far the mouse deviates from a given origin point. I think that the most sane origin point to go from is the center of your window. For each movement that the mouse makes from the center of the window, we need to:

Determine how much movement occurred on the x axis
Determine how much movement occurred on the y axis
Deaden this movement by a co-efficient to simulate mouse “sensitivity”
Set the yaw and pitch (head up/down, left/right) of the camera
Reset the mouse back to the origin point

Here’s how I’ve done it in code:

void mouseMotion(int x, int y) {
  // calculate the origin point (shifting right divides by two, remember!)
  int halfWidth = windowWidth >> 1;
  int halfHeight = windowHeight >> 1;

  // calculate how far we deviated from the origin point and deaden this
  // by a factor of 20
  float deltaX = (halfWidth - x) / 20.0f;
  float deltaY = (halfHeight - y) / 20.0f;

  // don't do anything if there wasn't any movement to report
  if ((deltaX == 0.0f) && (deltaY == 0.0f)) {
    return ;
  }

  // set the camera's orientation
  cam.yaw(deltaX);
  cam.pitch(deltaY);

  // reset the mouse pointer back to the origin point
  glutWarpPointer(halfWidth, halfHeight);
}

You can see that I’m using GLUT to do these demos. The only GLUT-specific piece of code here is the warp command which puts the mouse back onto the origin point. You should have an equivalent function to do the same in the framework of your choice.

Well, there you have it. You can now orient your camera using your mouse.

Older Newer

Cogs and Levers A blog full of technical stuff

Hadoop 2 (2.2.0) setup on Debian

Install Java

Add Hadoop Users and Groups

SSH Certificates

Download Hadoop

Setup Environment Variables

Configure Hadoop

Prepare the Filesystem

Start Hadoop

Getting started with Lua using C++

Introduction

Library Installation

Building applications

Test application

Installing SDL2 on Linux

Introduction

Dependencies

Compilation and Installation

Perlin Noise

Introduction

Understanding Noise

Follow up to a Camera Implementation

Introduction

The Idea