Dynamic module loading in Python

21 Jan 2016

The module system in fairly rich, where the import keyword is really only the tip of the iceberg. In today’s post, I’m going to go through a very simple dynamic module loading unit.

A module loader allows you to invoke pieces of code dynamically at run-time. Configurations of what gets executed when can be defined in a database, statically, where ever. What’s important is that we’ll reference our modules by name (using a string), have them loaded dynamically and then executed.

As a quick aside, my use of the term module throughout this article could refer to a python module; it could also refer to a small dynamic piece of code I’m calling a module.

Common module format

To get started, we really need to settle on a common format or structure that our modules will take. It’ll be this assumption that our host object will use to uniformly invoke these pieces of code. Immediately, this format has two major problems that need to be solved for this system to work. It needs to:

Provide a construction interface
Provide runnability

There are plenty of other things that we could add in:

Event for when the module is loaded and torn down
Top-level error handler for when exceptions bubble out of the module
Common logging framework

For the purposes of this article, we’ll focus on loading and executing the module.

What is a module?

For our implementation, we’re going to say that a module is a class. Making this decision to create a module as a class allows us to refer to our module as a definition (the actual class itself) and defer the instancing of our objects to the class’s construction invocation.

A very simple module might look like this:

class Module:

    def __init__(self, state):
        self.state = state
        
    def run(self):
        print "This is module1 doing its thing"

You can see that our constructor is taking in a parameter called state. There’s no real significance here aside from giving the module system the ability to send arbitrary state information during the instantiation process. The run function is what our host will be executing to perform the module’s work.

Factory construction

The factory pattern allows a developer to encapsulate the construction of related types inside of a function (or factory implementation class), and have this construction derived at run-time or configured elsewhere (think inversion-of-control). We’re going to borrow very shallowly from this concept.

Each module that participates on our framework must export a create function. This function can take parameters; so long as all of your factory constructors define the same interface it doesn’t matter. For today’s example, my create function takes an arbitrary object called state which just allows the implementing developer to send information to the constructed module:

def create(state):
    return Module(state)

The host

The host’s job is to:

Load a module (a python file) off disk
import it
Call the module factory create
Call the run method of the module

There’s a lot more fancy stuff that we could do of course. Going back to the start of this article, there are lots of different services that the host should be able to offer each module that gets loaded to provide the overall system a richer experience without re-writing common services. Logging, error handling, network and data connections could be simplified in the framework to provide a quick avenue to modules to be productive!

Our very simple host, would look something like this:

def run_module(module_name):
    name = "modules." + module_name
    mod = __import__(name, fromlist=[''])
    obj = mod.create({})
    obj.run()

run_module('mod1')

The heart of the host really is the run_module function and it leans heavily on the __import__ call to get its job done. The state parameter for the module is wasted in this context, but you can see how it’d be relatively easy to manage context aware state per process that you’re running.

The run method runs our module code.

Conclusion

This is just a simple, dynamic module loader. It can be applied in a lot of highly complex scenarios but the basic principles of keeping modules small, concise should help you not get ahead of yourself.

Convenience and performance with Python collections

17 Jan 2016

The collections library included with python has some very helpful utilities to make your programming life a little easier. In today’s post, I’m going to go through a few of them.

Named tuple

This is really the feature that brought my attention to this library, initially. Where a tuple is an immutable set of values that are unnamed, you can create a class using the namedtuple() function to bring a little more formality to your types:

import collections
Person = collections.namedtuple('Person', ['firstName', 'lastName', 'age'])
joe = Person("Joe", "Smith", 21)

# joe is now as follows
Person(firstName='Joe', lastName='Smith', age=21)

That’s a neat shortcut.

Counter

A Counter class is a dict object that when queried for a key that doesn’t exist, will return a 0; and create that item ready for counting.

animals = ['dog', 'cat', 'cat', 'bat', 'mouse', 'dog', 'elephant']
c = collections.Counter()

for animal in animals:
    c[animal] += 1

# "c" now looks like this
# Counter({'dog': 2, 'cat': 2, 'bat': 1, 'elephant': 1, 'mouse': 1})

Pretty handy.

deque

A basic stack or queue like data structure can be initialized with the use of the deque class. As this object does look a lot like a list, it’s important to remember why it exists:

Though list objects support similar operations, they are optimized for fast fixed-length operations and incur O(n) memory movement costs for pop(0) and insert(0, v) operations which change both the size and position of the underlying data representation.

This tells us that some internal implementation assumptions have been made to tailor the runtime usecase of this class specifically for statically sized queues.

orders = collections.deque()
orders.append({ 'name': 'Mario', 'pizza': 'Cheese' })
orders.append({ 'name': 'Joe', 'pizza': 'Supreme' })
orders.append({ 'name': 'Tony', 'pizza': 'Pepperoni' })

orders.popleft()
# {'name': 'Mario', 'pizza': 'Cheese'}

orders.popleft()
# {'name': 'Joe', 'pizza': 'Supreme'}

orders.popleft()
# {'name': 'Tony', 'pizza': 'Pepperoni'}

Scala

16 Jan 2016

Introduction

The Scala Programming Language is a language that brings together object oriented concepts with functional programming concepts on top of the jvm.

In today’s post, I’m going to go through some basic concepts of the language.

Classes

For our example, we’ll define a Player class. This will hold our player’s name, height and weight which won’t change once we set them. Note that we’re using the val keyword in the parameter list of the default constructor for the class. This automatically generates immutable members for us accessing this information.

class Player(val name: String, val height: Int, val weight: Int) {

  def getMessage(): String = "Game on!"

  def talk(): Unit = {
    val message = this.getMessage()
    println(s"$name the player says '$message'")
  }

}

We’ve also given our player the ability to talk. The player also has a message to say with getMessage.

Inheritance

We can inherit from this base Player class and define a Forward and a Back.

class Forward(name: String, height: Int, weight: Int) extends Player(name, height, weight) {
  override def getMessage(): String = "Uggg!"
}

class Back(name: String, height: Int, weight: Int) extends Player(name, height, weight) {
  override def getMessage(): String = "How does my hair look?"
}

Forwards and backs say different things, so we have overridden the default getMessage implementation in each case.

Traits

A trait is similar to the interface that you’d find in other languages. The main difference to a strict interface, is that a trait can have implementation. In the following example, the ValueEmitter trait is applied to different types of objects, but it commonly utilised to equate an answer.

trait ValueEmitter {
  def value(): Double
}

To represent a literal value and an operation both using this trait, we apply it to classes:

class LiteralValue(v: Double) extends ValueEmitter {
  def value(): Double = v
}

class Operation(val v1: ValueEmitter, val v2: ValueEmitter, val op: String) extends ValueEmitter {
  def value(): Double = {
    val left = v1.value()
    val right = v2.value()

    op match {
      case "+" => left + right
      case "-" => left - right
      case "*" => left * right
      case "/" => left / right
      case default => 0
    }
  }
}

Case Classes

Case classes allow you to concisely condense the definitions above:

abstract class ValueEmitter
case class LiteralValue(v: Double) extends ValueEmitter
case class Operation(v1: ValueEmitter, v2: ValueEmitter, op: String) extends ValueEmitter

This syncs up really well with the pattern matching ideas.

Pattern Matching

Following on with the example in Case Classes, we’ll write a function that uses pattern matching to ensure we’re getting the correct type through. Also see that we can pattern match on the values being passed through; not just the type.

def calculate(v: ValueEmitter): Double = v match {
  case LiteralValue(lv) => lv
  case Operation(v1, v2, "/") => {
    throw new Exception("I do not support divide")
  }
  case Operation(v1, v2, op) => {
    val left = calculate(v1)
    val right = calculate(v2)

    op match {
      case "+" => left + right
      case "-" => left - right
      case "*" => left * right
      case "/" => left / right
      case default => 0
    }

  }
}

Object

Static classes (or singleton objects) are just a way of defining single-use classes. You’ll see main sitting within an object definition rather than seeing main declared statically.

object Test {
  def main(args: Array[String]): Unit = {
    println("Hello, world")
  }
}

Another demonstrative use case for these objects is a configuration class:

object Config {
  def transactionDb(): String = "postgres://blah/xyz"
  def objectStoreDb(): String = "mongodb://quxx/abc"
}

Both transactionDb and objectStoreDb become accessible when prefixed with Config..

Accessors

You can short cut the creation of your accessors using your default constructor. As you’d expect, you use val for immutable, read-only properties and var for the read/write items.

class StockPrice(val code: String, var price: Double) {
}

The code on the stock doesn’t change but its price does.

These accessors can be defined manually using the following convention; this also allows you to specify any code that needs to execute within these accessors:

class StockPrice(val code: String, val initialPrice: Double) {

  private var _price: Double = initialPrice;

  def price: Double = _price
  def price_= (value: Double): Unit = {
    _price = value
  }

}

This is only a really small sample of the scala language, but will certainly get you up and running pretty quickly.

For more information, see the following links:

information_schema with PostgreSQL

10 Jan 2016

The information_schema in PostgreSQL holds a lot of really handy views with information about the current database. Very useful in investigation and discovery scenarios.

In today’s post, we’ll go through the tables that sit in this schema and how they can help. The information_schema documentation can be found here and is what this article has been based on.

Meta and Context

-- get the current database name
SELECT * FROM information_schema.information_schema_catalog_name;

-- what are the roles does the current user have that the admin option for?
SELECT * FROM information_schema.administrable_role_authorizations;

-- what roles are applicabl to the current user?
SELECT * FROM information_schema.applicable_roles;

-- attributes on composite data types that the current user has access to
SELECT * FROM information_schema.attributes;

Server

-- available character sets
SELECT * FROM information_schema.character_sets;

-- list all collations available to this database
SELECT * FROM information_schema.collations;

-- lists the available character sets that apply to the collations
SELECT * FROM information_schema.collation_character_set_applicability;

-- list all of the options defined for foreign-data wrappers
SELECT * FROM information_schema.foreign_data_wrapper_options;

-- list all foreign-data wrappers defined in the database
SELECT * FROM information_schema.foreign_data_wrappers;

-- lists all of the options defined for foreign servers in this database
SELECT * FROM information_schema.foreign_server_options

-- lists all of the standard sql features supported
SELECT * FROM information_schema.sql_features;

-- lists features that are implementation defined
SELECT * FROM information_schema.sql_implementation_info;

-- lists all of the sql languages supported
SELECT * FROM information_schema.sql_languages;

-- lists all of the sql defined feature packages are supported
SELECT * FROM information_schema.sql_packages;

-- lists all of the supported parts of the sql standard
SELECT * FROM information_schema.sql_parts;

-- lists the size limits in the database
SELECT * FROM information_schema.sql_sizing;

-- lists sizing profile information
SELECT * FROM information_schema.sql_sizing_profiles;

-- lists all of the foreign servers defined in this database
SELECT * FROM information_schema.foreign_servers;

-- lists all of the options defined for foreign tables in this database
SELECT * FROM information_schema.foreign_table_options;

-- lists all of the foreign tables 
SELECT * FROM information_schema.foreign_tables;

-- list all settings for user mappings
SELECT * FROM information_schema.user_mapping_options;

-- list all user mappings 
SELECT * FROM information_schema.user_mappings;

Catalog

-- list all check constraints
SELECT * FROM information_schema.check_constraints;

-- lists all of the parameters to functions in the database
SELECT * FROM information_schema.parameters;

-- lists all foreign keys in the database
SELECT * FROM information_schema.referential_constraints;

-- lists all of the functions in the database
SELECT * FROM information_schema.routines;

-- lists all of the sequences
SELECT * FROM information_schema.sequences;

-- lists all constraints from tables in this database
SELECT * FROM information_schema.table_constraints;

-- list all tables
SELECT * FROM information_schema.tables;

-- list all triggers
SELECT * FROM information_schema.triggers;

-- list all composite types
SELECT * FROM information_schema.user_defined_types;

-- lists all views in the database
SELECT * FROM information_schema.views;

-- list all transforms (9.5 ONLY)
SELECT * FROM information_schema.transforms;

Security and Privileges

-- list all columns and their priviledges
SELECT * FROM information_schema.column_privileges;

-- lists all privileges on columns
SELECT * FROM information_schema.role_column_grants;

-- lists all privileges on functions
SELECT * FROM information_schema.role_routine_grants;

-- lists all privileges on tables
SELECT * FROM information_schema.role_table_grants;

-- lists all privileges on udfs
SELECT * FROM information_schema.role_udt_grants;

-- lists all privileges on various objects 
SELECT * FROM information_schema.role_usage_grants;

-- lists all privileges on functions
SELECT * FROM information_schema.routine_privileges;

-- lists all of the table privileges 
SELECT * FROM information_schema.table_privileges;

-- list all udt privileges
SELECT * FROM information_schema.udt_privileges;

-- list privileges on various objects
SELECT * FROM information_schema.usage_privileges;

-- list all data types that the user has access to
SELECT * FROM information_schema.data_type_privileges;

-- list all enabled roles
SELECT * FROM information_schema.enabled_roles;

Explore

-- list all routines that are used by a check constraint
SELECT * FROM information_schema.check_constraint_routine_usage;

-- list columns using a domain defined inside of this database
SELECT * FROM information_schema.column_domain_usage;

-- list all columns that use types owned by the current user
SELECT * FROM information_schema.column_udt_usage;

-- list all columns used by constraints
SELECT * FROM information_schema.constraint_column_usage;

-- list all tables used by constraints
SELECT * FROM information_schema.constraint_table_usage;

-- list all domains based on data types owned by the current user
SELECT * FROM information_schema.domain_udt_usage;

-- lists all columns in the database restricted by primary,unique, foreign or check constraint
SELECT * FROM information_schema.key_column_usage;

-- list all columns that are used in views
SELECT * FROM information_schema.view_column_usage;

-- list all routines that are used in views
SELECT * FROM information_schema.view_routine_usage;

-- lists all tables that are used in views
SELECT * FROM information_schema.view_table_usage;

-- list all of the columns in the database
SELECT * FROM information_schema.columns;

-- list all triggers that specify update columns
SELECT * FROM information_schema.triggered_update_columns;

-- list options for any foreign table columns
SELECT * FROM information_schema.column_options;

-- list all constraints that belong to domains in the current database
SELECT * FROM information_schema.domain_constraints;

-- list all domains defined in the database
SELECT * FROM information_schema.domains

-- list all of the data types inside of array elements
SELECT * FROM information_schema.element_types;

-- lists all of the schemas 
SELECT * FROM information_schema.schemata;

Running X11 applications with port forwarding

07 Dec 2015

Today’s post is a quick tip on X11 port forwarding, and how to use it to run X11 applications remotely.

The setup

Your remote computer, the one that will actually run the application needs openssh installed. Use your favorite package manager to get that installed. You then need to edit your sshd configuration file to allow X11 port forwarding.

sudo emacs /etc/ssh/sshd_config

You need to make two edits to this file:

X11Forwarding     yes
X11UseLocalhost   no

Restart the ssh daemon.

Running

From your client computer now, connect to your remote host and run any X11 application that you want. It’ll appear on your client machine.

ssh -XC user@host
/usr/bin/firefox

Older Newer

Cogs and Levers A blog full of technical stuff