Archive for June, 2009

Linux.conf.au 2010 – Call for Papers opens

June 29th, 2009

The linux.conf.au (LCA) 2010 call for papers is now open! Woot! Woot! etc, etc.

Next year’s LCA is from Monday 18 January 2010 to Saturday 23 January 2010 in sunny downtown Wellington, New Zealand.

Speaking as a former organiser (AKA LCA Ghost) I think, in an entirely unbiased way, that it is the best Linux/ conference in the world. It covers everything from hardcore kernel hacking to discussions and legal issues. The best people come, the social events rock and the event is always well organised and friendly.

The CFP has opened today and closes the 24th of July, 2009 and its general blurb is:


The LCA2010 Papers Committee is looking for a broad range of papers spanning everything from programming and software to desktop and userspace to , government and education but there is one essential:

The core of your paper must relate to open source in some way, i.e., if it’s a paper about software then the software has to be licensed under an Open Source license.

The LCA2010 Papers Committee welcome proposals for Papers the following topics:

* Kernel and system topics such as filesystems and embedded devices
* Networking topics such as peer to peer networking, or tuning a TCP/IP stack
* Desktop topics such as office and productivity applications, mobile devices, peripherals, crypto & security and viruses and other malware
* Server topics such as clusters and other supercomputers, databases and grid computing
* Systems administration topics such as maintaining large numbers of machines and disaster recovery
* Programming topics such as software engineering practices and test driven development
* Free Software and Free Culture topics, including licencing and Free and Open approaches outside software
* Free Software usage topics, including home, IT, education, manufacturing, research and government usage.

Most presentations and tutorials will be technical in nature, but proposals for presentations other aspects of Free Software and Free Culture, such as educational and cultural aspects are welcome.

LCA2010 is pleased to invite proposals for three types of papers:

* Presentation – 45 minutes
* Tutorials – 1 hour and 45 minutes (short)
* Tutorials – 3 hours and 30 minutes (long)

Presentations are 45 minute slots (including questions) that are typically a one-way lecture from you to the audience – the typical conference presentation. These form the bulk of the available conference slots.

Tutorials are either 1 hour and 45 minutes, or 3 hours and 30 minutes in length, and work best when they are interactive or hands- in nature. Tutorials are expected to have a specific learning outcome for attendees.

To increase the number of people that can view your talk, LCA2010 may video the talks and make them publicly available after LCA2010. When submitting your proposal you will be asked whether materials relating to your paper can be released under a Creative Commons ShareALike License.

In conclusion – submit a paper – you won’t regret it.

Hudson and Amazon EC2 – the sequel

June 28th, 2009

A while back I talked about getting Hudson running with EC2-based slaves.  I had some issues – some self-inflicted like forgetting to add the IP address of the Hudson server to the appropriate EC2 Security Group to allow access port 22 – and some Hudson idiosyncrasies.  Now, however, I’ve got things running and am building Hudson jobs for Facter and Puppet.

One of the ironic key issues for us is that we’re testing a configuration management system but we can’t use that system because RUBYLIB issues mean that there is some bleed through between the installed Puppet version and the version being tested.  So instead the useful Hudson EC2 plug-in allows an init script to be run to prep each EC2 slave.  For each slave we run a simple set of commands that retrieves a script designed to bootstrap the host. In our case we do:

cd /tmp
wget http://pelin.lovedthanlost.net/hudson/debian/ec2-explode
chmod 0755 /tmp/ec2-explode
/tmp/ec2-explode

The ec2-explode script installs all the required packages and performs some needed Hudson set-up. I’ve got a bunch of init scripts for different platforms and the EC2 plug-in automatically installs a Java JDK after the init script is run. The slave is then ready to run the tests. And simple…

CIS Security Metrics Available

June 27th, 2009

The CIS has released a collection of metrics – CIS Security Metrics Guide (v. 1.0.0).  The project goal is to “develop a balanced combination of unambiguous and logically defensible outcome and practice metrics measuring” and to “utilize data commonly available in most enterprises.”

The following metrics are proposed and documented:

  • Application Security
    • Number of Applications
    • Percentage of Critical Applications
    • Risk Assessment Coverage
    • Security Testing Coverage
  • Configuration Change Management
    • Mean-Time to Complete Changes
    • Percent of Changes with Security Review
    • Percent of Changes with Security Exceptions
  • Financial
    • Information Security Budget as % of IT Budget
    • Information Security Budget Allocation
  • Incident Management
    • Mean-Time to Incident Discovery
    • Incident Rate
    • Percentage of Incidents Detected by Internal Controls
    • Mean-Time Between Security Incidents
    • Mean-Time to Recovery
  • Patch Management
    • Patch Policy Compliance
    • Patch Management Coverage
    • Mean-Time to Patch
  • Vulnerability Management
    • Vulnerability Scan Coverage
    • Percent of Systems Without Known Severe Vulnerabilities
    • Mean-Time to Mitigate Vulnerabilities
    • Number of Known Vulnerability Instance”

Download the metrics here or via direct PDF link.

Reductive Labs scores $2M for Puppet IT automation tool | VentureBeat

June 23rd, 2009

Now this is not a secret… :)

Reductive Labs scores $2M for Puppet IT automation tool | VentureBeat.

Congratulations to , Andrew, Teyo and everyone involved. Well deserved!

Chris Middendorp on Melbourne and Heroin

June 22nd, 2009

This is a great article , history and Melbourne.

OpenSourceBridge and Portland

June 20th, 2009

I am just back from OpenSourceBridge in Portland, Oregon.  I went to run a panel on configuration management that went pretty well – no fireworks which I think many were hoping for.  I also went to see Luke and shoot the breeze and talk Puppet stuff.

I also got to hang around Portland, one of my favourite cities, and visit some cool restaurants, bars, Powell’s – which is the world’s most rocking bookshop, and catch up with some friends.

All in all a great trip.  I took a handful of photos so I’ll post them all here.

Git … oh my

June 20th, 2009

In recent projects I’ve been using a lot of git and I love it.  As a distributed source control tool it’s brilliant.  This is particularly true when you need to gather and manage a wide variety of disparate patches and commits.  the projects I work we get patches via a lot of paths:

  • Diffs attached to tickets
  • Diffs sent via email
  • Git branches and cherry picks

With the former two paths we (mostly me) have been cutting and pasting patch diffs into files or using wget.  We then use the patch command to apply the diffs to various branches and then commit the results.

With this approach we often lose track of the patches ownership and author.  This is problematic from two perspectives – we can’t allocate credit where it is due and when something goes wrong with a commit it’s often hard to track down the original author.

We obviously don’t have this issue with Git repositories and merging branches or cherry picking specific commits.  With the latter it is easy to track patch authors and ownership – even through multiple merges and rebasing.  So I like it when people fork the Github repo and provide a commit or feature/ticket branch when supplying code.

But for the others I’m trying to bring our process a little closer to Git by using the git am and git apply commands (also git-am and git-apply – though the git-command syntax is being deprecated) to pull in diffs and patches.  The git am command processes mailboxes (mbox and Maildir), mail messages or RFC 2822 formatted messages provided via standard input.  The git apply command applies a unified diff file to the current working tree.

Let’s start with the easy to use git apply command.  Let’s take one of our use cases: downloading a patch from our tracker and applying it.

First, we wget our patch file:

$ cd /tmp
$ wget http://projects.redmine.com/project/tickets/2222/patch2222.diff

We then change directory into our Git repo and check what branch we’re in.

$ cd ~/src/puppet
$ git branch
* master

We can now feed our patch file into git apply using something like cat and a pipe.

$ cat /tmp/patch2222.diff | git apply

This will add the patch to the current branch and commit it.

Another useful trick is the –amend option.  If your current commit is not yet pushed you can amend it.  Just make your required edits, git add the updated files:

$ git add filename

And then run:

$ git commit --amend

Git will populate your editor with the last commit message and you can update the commit.

We could also create a separate branch for our new commit.

$ git checkout -b tickets/master/2222
$ git apply < /tmp/patch2222.diff

Here we’ve directed the file straight into git apply rather than piped the output of the cat command.

We can then merge this into the appropriate branch when ready.

$ git merge tickets/master/2222

Don’t forget you can also use the git rebase command to ensure your branch is rebased against the branch you’re merging into, to squash multiple commits or to redo the commits – git rebase (especially the –interactive switch) is the business. :)

Using our second command, git am, we can also pull patches straight from a mail client.  For example, Thunderbird, I open up the message I want to import then ( OSX – PC it’s basically the same commands prefixed with Ctrl not Command):

  1. Command-U to show the full message including headers
  2. Command-A to select all of the message
  3. Command-C to copy the selection

I can then go to the command line and do:

$ cat | git am

Followed by a Command-V to paste the content, and then Control-D to end the cat and submit my patch to be committed.

The git am command is quite clever.  The author of the commit will be pulled from the From line of the message, the date and time of the commit from the date and time the message was sent and the Subject and Body are used as the title and body of the commit message.

We can also add the -s switch to the git am command.  This adds a “Signed-off-by” line to the commit message using your details (usually name and email address).

$ cat | git am -s

It’s not perfect model yet – I probably should be using git am a Maildir or mbox file which contains our patches but in our small development team I have the luxury of just selecting the patches and emails I want.

Hudson and Amazon EC2

June 8th, 2009

So my biggest gripe with Hudson is slave nodes. In Puppet land we need to run our tests a wide variety of platforms – it’s a system/configuration management tool that runs just about every flavour of *nix (and soon to be Windows) around: Linux (a bucket load of distros), *BSD, OSX, AIX, Solaris, HP-UX, amongst others. We need to ensure it builds, runs and configures things these platforms and that new features and functions don’t break things. To do this there is a huge management overhead – especially as build/test is generally two or three people – mostly and I.

In this process I’ve had many struggles with getting people to submit slaves and with managing my own slave node network for Hudson. It’s bloody annoying to have to fight RUBYLIB issues, Gem versions, installed versions of Puppet and Facter and a dozen other issues with running tests slaves before you can even identify and fix a failed test.

I finally threw in the towel and decided to look at some other CI engines – perhaps my initial look and selection of Hudson had been premature.  But another review and installation of a wide variety of tools suggested to me that Hudson was the right choice. So an impasse.

Then along came the Hudson Amazon EC2 plug-in.  It allows Hudson to run up Amazon EC2 instances when required as slaves.  This means with a few judicious choices of AMIs I can quickly run up a test farm that covers all my requirements – Linux, Solaris, even Windows when we merge in the new Windows code (0.25.0beta2 I hope).  It’s not quite working yet – for reasons that aren’t 100% clear to me yet. :)   But it’s the right path and I hope it’s going to make life much easier.  More how it all works … when it all works..

UPDATE:  It now works – mostly a PEBCAK issue – the plug-in requires that the EC2 Security Group is configured to allow a connection port 22 from the Hudson master.  Next steps some proper implementation… :P

New home page

June 6th, 2009

The wonderful and highly talented Erin Knowles designed me a new home page – bio, CV, , etc – that is very, very cool.  It looks very speck and is nice and simple.  I like nice and simple.

Erin also makes funny little things called efki star creatures which would seem to make perfect gifts for kiddies.

P.S.  This doesn’t mean this page goes away – that’s my “this is me the author page and the projects I work ” page.  This page stays as the “this is my blog page” etc.

Puppet Extras …. denizens of the Puppet ext directory

June 2nd, 2009

I’ve been meaning to do this for ages….  In the Puppet tarball is a
directory called “ext” that contains some extras that aren’t exactly
mainline but could be useful to people.  You can see it’s current
contents here.

So the “ext” directory has got a number of useful tools, scripts and
add-ons including:

dqfix.sql – A script to remove duplicate resources from a MySQL Stored
Configuration database

emacs – An emacs mode for editing Puppet manifests

ldap – The Puppet LDAP schema

logcheck – Puppet configuration for logcheck

nagios/check_puppet.rb – A Nagios plug-in for monitoring Puppet

nagios/naggen – An application to generate Nagios configuration from a
Stored Configuration database.

module_puppet – Runs a puppet script as a cfengine module

puppetlast – Command-line tool to checks the status of your clients and
when they last connected

puppetlisten – A daemon to trigger Puppet runs clients

puppetstoredconfigclean.rb – a script to clean a host’s data from a
stored configuration database

puppet-test – A script to measure Puppet client performance

rack – A Puppet module for configuring Puppet with Passenger/Rack/mod_rails

vim – Syntax highlighting and manifest editing support for Vim

P.S.  These are offered without warranty and we make no guarantees that
they all work.