Python Plugins (Dynamic Modules)

Programming Comments Off

Dec 032010

Introduction

Sometimes dynamically loaded modules (plugins or extensions) are pretty convenient to provide extensible functionality from your applications. For example, you need to provide a command that provides known data sources to subcommands but want the subcommands to be easily written and added even after the application has been finalized. We could do this with a simply proper modular design but it seems more natural to allow for the subcommands to be defined elsewhere with a standard interface to allow for extensible behavior even after the initial application development cycle.

The Problem

How do we find and then load and then run code that we didn’t necessarily write?

The first step is fairly obvious we simply ask (via a parameter, config option, or other method) where the code that should be loaded is located. Once we have that the other steps are much easier. In more detail, we need to know a location for code that follows our plugin API resides. To do this we can use the following code (where d is the directory with our plugins):

[sourcecode language="python"]
sys.path.append(d)
files = itertools.chain(*[ [ os.path.join(x[0], fs) for fs in x[2] ] for x in os.walk(d) ] )
plugins = [ f.split('/')[-1].split(‘.’)[0] for f in files if f.endswith(‘.py’) ]
modules = [ __import__(p, globals(), locals(), [], -1) for p in plugins ]

for p,m in zip(plugins,modules):
matches = [ x for x in m.__dict__.keys() if x.lower() == p ]
if len(matches) == 1: # and issubclass(m.__dict__[matches[0]], CorkyCommand):
self._commands.append(m.__dict__[matches[0]]())
[/sourcecode]

Break Down

Add our directory to the python module path so we can simply load them by name
Get a list of the files in this directory
Filter this down to the names of the python files to find the Class that we need to create an instance of
Import the modules as module objects we can manipulate
Loop through the correlated list of plugin names and module objects
Look for an object in the module dictionary that matches the name of the file (case insensitive)
Find a match we then add an instantiated object of the class we found

Quite a bit is going in this short snippet of code but the important thing is it takes a directory path and creates a list of instantiated plugin objects we can use just like any other object variable. Once we have the objects it’s simply a matter of calling functions on them: `self._commands[n].method()`.

Conclusion

Getting a modular design can be daunting and making that modular design as dynamic as possible can be even more daunting but the modern languages (this technique but not syntax works with ruby as well) make this process much easier than the compiled languages (More to come on that later I hope).

Pclean, a new /etc/portage/package.* cleaner

Linux Guides Comments Off

Nov 072010

I’ve often gotten frustrated with my /etc/portage/package.* files when they become massive and full of crud that I don’t even have installed any longer. Because of this I have crafted a simple little utility to clean out packages that are no longer installed and use flags that are no longer valid from these files. This should help trim the cruft from the Gentoo configuration.

The utility, pclean, does all of this and only has one major problem (so far) before I shall call it good enough for a 1.0 release. If you would like to try this little utility; it’s available in my overlay and if you notice any odd behavior please report it to my bugzilla.

Cacti Ping NaN With iputils-20100418

Linux Guides Comments Off

Oct 242010

Introduction

It appears the output of ping has changed in this release of iputils from referring to the icmp sequence numbers as icmp_seq to icmp_req which obliterates the ping.pl script that cacti uses to do pings of servers it watches.

The Fix

The fix is quite simple: change the seq to req in the grep line of ping.pl but the following fix is probably more versatile (and will be checked for upstream).

*** ping.pl     2010-07-09 17:33:46.000000000 -0500
--- ping.pl.new 2010-10-24 18:22:16.325881546 -0500
***************
*** 4,10 ****
  $host = $ARGV[0];
  $host =~ s/tcp:/$1/gis;

! open(PROCESS, "ping -c 1 $host | grep icmp_seq | grep time |");
  $ping = ;
  close(PROCESS);
  $ping =~ m/(.*time=)(.*) (ms|usec)/;
--- 4,10 ----
  $host = $ARGV[0];
  $host =~ s/tcp:/$1/gis;

! open(PROCESS, "ping -c 1 $host | grep -E icmp_\(r\|s\)eq | grep time |");
  $ping = ;
  close(PROCESS);
  $ping =~ m/(.*time=)(.*) (ms|usec)/;

Conclusion

Sometimes things break just because of small changes. This is a simple example of that and the quick fix for the annoyance of not recording your ping times.

Hamachi on Gentoo

Linux Guides Comments Off

Sep 272010

Introduction

Hamachi is a lightweight personal VPN connector that is a breeze to setup but there can be some pain if you don’t know what to expect. As always Gentoo provides us with an ebuild that simplifies the installation process but getting up and running is still a little confusing.

Installation and Setup

User Specific Configurations

The obvious first step is `emerge -av hamachi` (this only available to ~arch right now so add to `package.keywords` as necessary). The following are the typical instructions to install hamachi on Gentoo from portage:

Add to `packages.keywords` if necessary
`emerge -av hamachi`
`rc-update add tuncfg default`

After these steps have been taken you can run hamachi as any user on the system for ad-hoc VPN creation.

Server Wide Configuration

If you prefer to do a system wide on boot VPN with hamachi this is also possible but requires a slightly different setup:

Add to `packages.keywords` if necessary
`emerge -av hamachi`
rc-update add hamachi default

Now all configuration should be placed inside `/etc/hamachi` for this setup so the system will automatically start routing traffic correctly.

Kernel Configuration

For hamachi to work correctly you do need the tun parameter in your kernel or loaded as a module. This parameter is located in Device Drivers->Network device support->Universal TUN/TAP device driver support.

Using Hamachi

Now that hamachi is on the system we need to start using it. The server-wide installation doesn’t require this (but I’m sure you can use this method to create a configuration usable by the server-wide instance) but the user specific usage does.

Starting hamachi is as simple as the following:

`hamachi-init`
`hamachi start`
`hamachi login`
`hamachi create [ ]`
`hamachi join [ ]`
`hamachi go-online `

That’s it. You’re now connected to a private network named . You can view who else is connected to your network with `hamachi list` and `hamachi get-nicks`.

Conclusion

Setting up a VPN can be daunting (see the OpenVPN configuration documentation) or it can be a breeze with hamachi. Need a quick VPN for LAN gaming or a VPN for performing maintenance over the internet on a device behind a firewall? Hamachi may be the quick solution you’re looking for.

Bind Mount and Uses to Avoid

Linux Guides Comments Off

Sep 192010

Introduction

Not overly recently, I was asked to configure a chroot jail by setting up bind mounts instead of copying over the correct binaries. The incident caused a really interesting system failure, a wake-up call to myself, and an ambitious project to create an autochroot command (more later).

The Back Story

When setting up a chroot jail it’s well known that all the binaries and references that will be used in that jail need to be inside that jail. This creates quite a complicated configuration for something even as simple as creating a chroot jail that only has bash and bash internals in it. Not only do we need to create a clean directory structure and duplicate our file system hierarchy but we also need to create all the dependencies (and for bash this quickly grows):

ldd /bin/bash
linux-vdso.so.1 => (0x00007fff189ff000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00007f6ca8604000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f6ca8400000)
libc.so.6 => /lib/libc.so.6 (0x00007f6ca809a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6ca8854000)

Once all the dependencies are in place, the proper items to do work in this minimal shell environment also need to be installed. This may include device nodes or system environment files (e.g. /etc/passwd, /etc/shadow). Which make some commands a real pain to install in a chroot by hand (e.g. /usr/bin/ssh, /usr/bin/scp, etc).

The Problem

So, it’s hard; that’s not a big deal. There are ways to make this easier. Things like bind mounts:

mount -o bind

These handy features allow you to take any directory in the system and mount it somewhere else. These become extremely useful when setting up a chroot environment for system recovery via a livecd but can be a ticking time bomb if used incorrectly.

Alright, let’s solve our chroot problem with bind mounts, but I bet you can guess by this point how it’s going to be accomplished. We’re simply going to mount all of the necessary areas of the external filesystem inside the chroot:

mount -o bind /dev /chroot/dev
mount -o bind /lib /chroot/lib
mount -o bind /usr/lib /chroot/usr/lib

Now, that chroot was much easier to set up and work with, but what happens if an unsuspecting party does the unthinkable: `rm -fr /chroot`. They’re simply cleaning up a chroot environment that doesn’t need to be used anymore, right? Unfortunately, that’s not the case. This will remove all files in /chroot including /dev, /lib, /usr/lib, etc.

The Lesson

Don’t ever use bind mounts in chroot environments unless you know exactly what you are doing! Even then, it may be best practice to simply not use bind mounts. You may be thinking, why can’t we simply mount these points readonly? Unfortunately, bind mounts aren’t that smart. This is what happens when you attempt a bind mount readonly:

mount -o bind,or /proc /mnt
mount: warning: /mnt seems to be mounted read-write.
mount
/proc on /mnt type none (rw,bind)

Not quite what we were looking for and unfortunate for our quick solution.

Conclusion

Never use bind mounts for a chroot environment (except maybe livecd chroot environments). It may take more time but copying the necessary binaries to the chroot is much safer and will keep your system from any inadvertent harm.

Now, I mentioned that I came up with a much more ambitious solution to this problem, autoroot. This small utility (when finished) will be able to take a directory and list of binaries and automatically create an appropriate chroot environment. The system should be smart enough to take into consideration different distributions and installation points (a current limitation of existing solutions).

Older Entries Newer Entries

February 2012
S	M	T	W	T	F	S
« May
	1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29

Suffusion theme by Sayontan Sinha

Alunduil's Hosting

Python Plugins (Dynamic Modules)

Introduction

The Problem

Break Down

Conclusion

Pclean, a new /etc/portage/package.* cleaner

Cacti Ping NaN With iputils-20100418

Introduction

The Fix

Conclusion

Hamachi on Gentoo

Introduction

Installation and Setup

User Specific Configurations

Server Wide Configuration

Kernel Configuration

Using Hamachi

Conclusion

Bind Mount and Uses to Avoid

Introduction

The Back Story

The Problem

The Lesson

Conclusion

Archives

Alunduil's Hosting

Python Plugins (Dynamic Modules)

Introduction

The Problem

Break Down

Conclusion

Pclean, a new /etc/portage/package.* cleaner

Cacti Ping NaN With iputils-20100418

Introduction

The Fix

Conclusion

Hamachi on Gentoo

Introduction

Installation and Setup

User Specific Configurations

Server Wide Configuration

Kernel Configuration

Using Hamachi

Conclusion

Bind Mount and Uses to Avoid

Introduction

The Back Story

The Problem

The Lesson

Conclusion

Common Tags

Archives