Cogs and Levers A blog full of technical stuff

Custom Installs with Checkinstall

Introduction

Sometimes it’s necessary to install a piece of software from source. This is normally an awkward process when you’ve had a package manager taking care of all your software needs when you’re faced with the proposition of installing something that the package manager is unaware of. Another concern is that some software developers don’t do the simple things well - some makefiles won’t even offer you the ability to uninstall a piece of software leaving you to try to remove the files that have been peppered into your system directories.

In today’s post, I’ll walk through a sample usage of the application CheckInstall for Debian based Linux distributions.

What is it?

From the CheckInstall page on the Debian Wiki:

CheckInstall keeps track of all the files created or modified by your installation script (“make” “make install” “make install_modules”, “setup”, etc), builds a standard binary package and installs it in your system giving you the ability to uninstall it with your distribution’s standard package management utilities.

The thing I love about the Debian distribution is the stability of the packages in its repositories. Sometimes, it’s also what I’m not to fond of as software vendors are bringing out new versions of their software and they don’t make it into the stable repositories until they’re deemed stable (which takes ages!) or they may never make it into the repositories.

Using CheckInstall

I’ve used CheckInstall for quite a few packages in the past. Just recently, I’ve used it to manage the installation of SDL2 onto my system.

# extract your source package
$ tar -zxvf SDL2-2.0.1.tar.gz

# configure and build as usual
$ cd SDL2-2.0.1
$ ./configure
$ make

# use checkinstall on the installation step
$ sudo checkinstall make install

After this process had finished, a deb file was created for me which represented the files that had been installed on the system and the deb itself had been applied to the system.

The idea here is that it simplifies re-installation and removal by proxy of the generated deb package.

Hadoop Links

Apache Hadoop NextGen MapReduce (YARN)

File System Shell Guide

YAHOO! Hadoop Tutorial

Hadoop 2 (2.2.0) setup on Debian

Today’s post will just be a walk through of the steps required to install Hadoop 2 on Debian Linux. Please note that this is for a single node installation only. This guide is heavily based on the Ubuntu instructions found here.

Install Java

	
# install the java jdk
$ sudo apt-get install openjdk-7-jdk
 
# make a jdk symlink
$ cd /usr/lib/jvm
$ ln -s java-7-openjdk-amd64 jdk
 
# make sure that ssh server is installed
$ sudo apt-get install openssh-server

Add Hadoop Users and Groups

	
# create a new group for hadoop
$ sudo addgroup hadoop
 
# create the hduser and put them in the hadoop group
$ sudo adduser --ingroup hadoop hduser
 
# add them to the sudo group also
$ sudo adduser hduser sudo

Now login as “hduser”.

SSH Certificates

	
# generate your key
$ ssh-keygen -t rsa -P ''
 
# set your public key as authorized
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
 
# test out ssh
$ ssh localhost

Download Hadoop

	
# downoad the package
$ cd ~
$ wget http://mirror.rackcentral.com.au/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
 
# extract the package
$ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local
$ cd /usr/local
$ sudo mv hadoop-2.2.0 hadoop
 
# get the hduser to take ownership
$ sudo chown -R hduser:hadoop hadoop

Setup Environment Variables

Add the following lines to your ~/.bashrc

# Hadoop variables
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL

Add the following lines to /usr/local/hadoop/etc/hadoop/hadoop-env.sh

# modify JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/jdk/

Re-login to your machine as hduser, and check the hadoop version.

$ hadoop version

Configure Hadoop

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/core-site.xml

<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/yarn-site.xml

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Make a copy of the mapred-site template file

	
$ mv mapred-site.xml.template mapred-site.xml
$ vi mapred-site.xml

Add the following lines into the <configuration> node within /usr/local/hadoop/etc/hadoop/mapred-site.xml

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

Prepare the Filesystem

	
# create the physical directories
$ cd ~
$ mkdir -p mydata/hdfs/namenode
$ mkdir -p mydata/hdfs/datanode

Add the following lines into the <configuration> node /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/mydata/hdfs/datanode</value>
 </property>

Format the namenode

	
$ hdfs namenode -format

Start Hadoop

	
$ start-dfs.sh
$ start-yarn.sh
 
# check that services are running
$ jps

Run the Example

	
$ cd /usr/local/hadoop
$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

Getting started with Lua using C++

Introduction

Lua is a programming language that has seen increased popularity from the game development industry. It’s put to use in the disciplines of providing configuration data all the way through to scripting automated character actions.

In today’s post, I’ll walk you through the setup process of the libraries up to writing your first testing application within a Linux environment.

Library Installation

Before you get downloading and building this library, you’ll just need to ensure that you have a build environment installed and are able to compile.

At the time of writing this article, the Lua website had 5.2.3 as their latest release. From their downloads page, grab the latest tar.gz archive and extract it onto your machine. Following along with their building instructions, issuing “make linux test” whilst in the extracted folder did the trick. It built Lua ready for me to use. A nice convenience of the make file was the “make local” option. Upon issuing this request, the make system will prepare an installation folder that is suitable for you to use locally (i.e. not installed to the system).

When it comes to downloading the latest versions of libraries, I’ll rarely install these to my system. Rather, I drag them around for each project that needs them so that the project determines its dependencies as opposed to my system.

From here, I prepare a distributable directory of development files that I know that each of my project needs. In the case of Lua, I create the following structure:

.
└── lua-5.2.3
    ├── include
    │   ├── lauxlib.h
    │   ├── luaconf.h
    │   ├── lua.h
    │   ├── lua.hpp
    │	  └── lualib.h
    └── lib
    	├── liblua.a
    	└── lua
    		└── 5.2

I have all of the development headers available (under the “include” folder) and a static version of the Lua library (under lib).

Building applications

When building Lua applications, you’ll need to specify the libraries and include folders to your compiler so it knows where to find them. For a test application that I’d written, the following command line compiled an application for me without any trouble.

$ g++ -Ilib/lua-5.2.3/include -Llib/lua-5.2.3/lib/ -llua -ldl

You can see at the end there, mention of both the “lua” and “dl” libraries.

Test application

A very simple test to start will be creating a program that will execute a Lua script, pull out on of its global variables and display it to screen.

Here’s our Lua script:

x = 10

Pretty simple. We have one variable x set to 10. Now here’s the C++ code that we use to read that one variable out and present it to screen.

#include <iostream>
#include <lua.hpp>

int main(int argc, char *argv[]) {
  // create a new lua context to work with
  lua_State *L = luaL_newstate();

  // open any library we may use
  luaL_openlibs(L);

  // read the Lua script off disk and execute it
  if ((luaL_dofile(L, "test.lua")) != 0) {
    
    // handle any errors 
    std::cout << "unable to load test.lua" << std::endl;
    return 1;
    
  }

  // put the value of X at the top of the stack
  lua_getglobal(L, "x");
  
  // interpret the value at the top of the stack 
  // as an integer and put it in the variable "val"
  int val = (int)lua_tointeger(L, -1);
  
  // pop the value of X off the stack
  lua_pop(L, 1);

  // write the value out
  std::cout << "Value of X: " << val << std::endl;

  // finish up with the Lua context
  lua_close(L);

  return 0;

}

I think that the source code above (paired with the documentation on the Lua website) should make things pretty straight forward.

That’s it for today. This is only scratching the surface on what Lua can do. For my purposes right now, I just need to feed configuration values into programs, this fits the purpose really well.

Installing SDL2 on Linux

Introduction

SDL2 was released a little while ago, but still hasn’t made it into the stable repositories of some Linux distributions. After doing a big of digging, it’s not too hard to get this installed yourself - most of the advice offered in this post comes from an answer on the Ubuntu forums here.

In today’s post, we’ll install SDL2 on a Debian/Ubuntu style distribution from source.

Dependencies

First thing before we download and compile the SDL2 source is to get some of the dependencies installed on your system. The following install line will put all of the libraries that SDL2 requires:

$ sudo apt-get install build-essential xorg-dev libudev-dev libts-dev libgl1-mesa-dev libglu1-mesa-dev libasound2-dev libpulse-dev libopenal-dev libogg-dev libvorbis-dev libaudiofile-dev libpng12-dev libfreetype6-dev libusb-dev libdbus-1-dev zlib1g-dev libdirectfb-dev

Once all of these have installed successfully, you’ll need to download a copy of the source. All downloads can be found here. This guide will assume that you’ll download the .tar.gz source archive.

Compilation and Installation

Extract the package into a directory under your home directory, somewhere . . .

	
$ tar -xzf SDL2-*.tar.gz
$ cd SDL2-*

Next we’ll configure, build and install the libraries.

	
$ ./configure
$ make
$ sudo make install

Once compilation and installation have complete, you’ll need to update your library links/cache. Do this with the following command:

	
$ sudo ldconfig

That’s all there is to it.