Posts Tagged ‘repo’

Puppet Module Repository isn’t just for modules

June 1st, 2010

You can store more than just your modules at the Forge. :)   I just added my types and providers to my collection of modules at the new Puppet Module Forge.  I’d love to all those people maintaining types and providers, functions, and facts add theirs to the Forge also.  It’s a cool way to share your code (and the site allows you to provide links back to your code and ticketing system so user’s can report bugs).  In time I hope most people’s environments will consist of the core types and providers bundled with Puppet and a selection of cool code generated by the community and sourced from the Puppet Forge.

No comments »

Posted in Blog, devops, open source, puppet

Tags:

Puppet Forge in beta!

May 27th, 2010

The Puppet Forge AKA the Puppet Module Repository live and operational.  It’s a store of Puppet modules (and types and providers) that allows you to share your awesome code and modules with others.

It also comes with the puppet-module tool that allows you to build modules for, manage and install modules from the forge.  You can install puppet-module via a gem:

$ sudo gem install puppet-module

Both the site and tool are in public beta right now so hammer away at it and tell us what you think!

1 comment »

Posted in Blog, devops, open source, puppet, rails

Tags:

Creating Puppet types and providers is easy…

February 1st, 2010

Puppet types are used to manage individual configuration items.  Puppet has a package type, a service type, a user type, etc.  Each type has providers. Each provider handles the management of that configuration a different platform or tool, for example the package type has aptitude, yum, RPM, and DMG providers (amongst 22 others – what wrong with people that they need to invent new packaging systems… but I digress).

There are a lot of types, in fact I think Puppet covers a pretty good spectrum of configuration items that need to be managed.  I don’t know of anything in particular that missing that I can’t live without.  But there are little gaps that are annoying, I’d like network and firewall types for example, but creating both these types in a generic enough way to support multiple platforms would be, IMHO, a non-trivial problem. 

Another gap VCS/DVCS management. A lot of people use source code in repositories to do things with (including install stuff from you bad people – package things … it’s healthier). Puppet currently relies creating and removing these repositories with the exec type (which executes scripts or binaries), for example:

exec { "svn co http://core.svn.wordpress.org/trunk/ /var/www/wp":
    creates => "/var/www/wp",
}

This a bit ugly and it’d be a lot easier to write a Puppet type to manage repositories. But Puppet types and providers are written in Ruby and really, really complex and hard to develop. Right? Right?

No. No, they are not… and I’m going to create a simple type and provider to show you. :)

Here’s a very (very!) simple Puppet type, called , for managing repositories. I’ve created providers for and Git as examples also. The first part of the type the type itself – these are usually stored in lib/puppet/type or distributed via modules (see the PluginsInModules page in the Puppet wiki). I’ll create a file called .rb.

$ touch repo.rb

And then populate the file:

Puppet::Type.newtype(:repo) do
    @doc = "Manage repos"
 
    ensurable
 
    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
 
    end
 
    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end
end

So – pretty simple. We create a block Puppet::Type.newtype(:) do that creates a new type, which we’ve called .

Inside the block we’ve got a @doc string. This the documentation for the type. Add whatever level of detail and examples in here that required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. Puppet types use the ensure property to determine the state of a configuration item.

service { "sshd":
    ensure => present,
}

The ensurable statement tells Puppet to expect three methods: create, destroy and exists? in our provider. These methods, allow, respectively:

  • A command to create the resource
  • A command to delete the resource, and
  • A command to check for the existence of the resource

All we then need to do specify these methods and their contents and Puppet creates the supporting infrastructure around them but more this when we look at our providers.

Next, we’ve defined a new parameter – this one called source.

    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
    end

The source parameter will tell the type where to go to retrieve/clone/checkout our source .

In this parameter we’re also using a hook called validate. Normally used to check the value for appropriateness here we’re using it to take a guess at what provider to use. Our code says, if the source parameter starts with git then use the Git provider, if not default to the Subversion provider. This obviously fairly crude as a default and we can override this by defining the provider attribute in our resources:

provider => git,

We’ve also used another piece of Puppet automagic, isnamevar, to make this parameter the “name” variable for this type. In Puppet-speak, the value of this parameter used as the name of the resource.

(Types have two kinds of values – properties and parameters. Properties “do things”. They tell us HOW the provider works. We’ve only defined one property, ensure, by using the ensurable statement. Parameters are more like variables, they contain information relevant to configuring the resource the type manages rather than “doing things”.)

Finally, we’ve defined another parameter, path.

    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end

This a variable value that specifies where the type should put the cloned/checked-out . In this parameter we’ve again used the validate hook to create a block that checks the value for appropriateness. Here we’re just checking, very crudely, to make sure it looks like the destination path a valid fully-qualified file path. We could also use this validation for the source parameter to confirm a valid source URL/location was being provided.

(You can also use another hook called munge to adjust the value of the parameter rather than validating it before passing it to the provider.)

And that it for the type.

Next, we need to create a provider for our type. Let’s start with a Subversion provider like so:

require 'fileutils'
 
Puppet::Type.type(:repo).provide(:svn) do
    desc "SVN Support"
 
    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"
 
    def create
        svncmd "checkout", resource[:name], resource[:path]
    end
 
    def destroy
        FileUtils.rm_rf resource[:path]
    end
 
    def exists?
        File.directory? resource[:path]
    end
end

Up front we’ve required the fileutils library, which we’re going to use a method from. Next, we’ve defined the provider as a block:

Puppet::Type.type(:repo).provide(:svn) do

We tell Puppet that this a provider called for the type called .

Then we use a desc method that allows us to add some documentation to our provider.

Next, we define the commands that this provider will use, here the and svnadmin binaries, to manipulate our resource’s configuration.

    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"

Puppet uses these commands to determine if the provider appropriate to use a client, if Puppet can’t find these commands in the local path then it will disable the provider.

Next, we’ve defined three methods – create, destroy and exists?. Sounds familiar? Yep, these are the methods that the ensurable statement expects to find in the provider:

The create method ensures our resource created. It uses the command to create a with a source of resource[:name] (remember the source parameter in our type also the name variable of the type – we could also specify resource[:source] here too) and a destination of resource[:path] (the value of the path attribute).

The delete method ensures the deletion of the resource. In this case, it deletes the directory and files specified in the resource[:path] parameter.

Lastly, the exists? method checks to see if the resource exists. Its operation pretty simple and closely linked with the value of the ensure attribute in the resource:

  • If exists? false and ensure present, then create method will be called.
  • If exists? true and ensure set to absent, then the destroy method will be called.

In this case the exists? method checks if there already a directory at the location specified in the resource[:path] parameter.

So, let’s put all this together and create a resource with our new type. I’ve assumed you’ve already distributed your type and providers to Puppet. We can then create a resource like:

repo { "wp":
    source => "http://core.svn.wordpress.org/trunk/",
    path => "/var/www/wp",
    ensure => present,
}

Simple eh? We specify a resource, the source we wish to check out or clone from, the destination path and the ensure attribute (present or absent) and that’s it.

You can see the complete code for this type and its providers at my Puppet repository on GitHub. It’s obviously very basic but should be easy to extend to provide additional capabilities (and currently has no tests – my bad). You can find further documentation (in a lot more detail!) creating your own types and providers at the Puppet wiki.

14 comments »

Posted in Blog, open source, puppet

Tags:

Puppet’s BuildBot

August 24th, 2008

So rather than doing the work I actually should be I’ve been playing with BuildBot. I had intended to get around to setting up BuildBot sometime in the next couple of months but I got hooked.

The reason I wanted to have a look at BuildBot was that Puppet has reached a stage where we simply can’t test every platform it runs . We are also starting to get patches from a wider variety of sources. Buildbot will allow us to execute our tests a wider variety of platforms. Hopefully with the cooperation of the community we can gather a really big collection of build platforms to test .

Here’s the blurb for BuildBot

The BuildBot a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.

The overall goal to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin.

It’s a very easy tool to deploy. The hardest part has been the slightly broken Git source handling and the assumption that any Git local. I need to have a local Git to allow BuildBot to submit the right commits references to the PBChangeSource function.

But I designed a basic process for handling new commits:

1. Commit pushed to .
2. Commit bot at picks up commit and sends it to BuildBot Master.
3. BuildBot uses the git_buildbot.py script to calculate the before/after commit and branch references and tell BuildBot about them.
4. BuildBot executes the build and tells each slave to retrieve the commit and runs the tests. Currently we’re running:

a. All the Unit tests
b. All the tests

5. We then get the results of the tests the website and in an email to the new Puppet Builds mailing list.

In addition I’ve also enabled BuildBot’s IRC bot and added a new bot, called pinocchio, to the #puppet channel that reports build status.

At this stage it’s all in test mode and when I’ve ironed out a few issues we should be in a position to do a production installation at ReductiveLabs and start canvassing for build slaves.

UPDATE

After mucking around with Buildbot I just couldn’t get a whole bunch of issues with Git resolved so we changed to Hudson as our CI – which works much better.  The message overall – CI and Git: still a young pair.  I’ve included our configuration below for edification:

# -*- python -*-
# ex: set syntax=python:

# This a sample buildmaster config file. It must be installed as
# ‘master.cfg’ in your buildmaster’s base directory (although the filename
# can be changed with the –basedir option to ‘mktap buildbot master’).

# It has one job: define a dictionary named BuildmasterConfig. This
# dictionary has a variety of keys to control different aspects of the
# buildmaster. They are documented in docs/config.xhtml .

# This the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### BUILDSLAVES

# the ‘slaves’ list defines the set of allowable buildslaves. Each element
# a tuple of bot-name and bot-password. These correspond to values given to
# the buildslave’s mktap invocation.
from buildbot.buildslave import BuildSlave

c['slaves'] = [BuildSlave("debian", "debian"),
BuildSlave("freebsd", "freebsd"),
BuildSlave("redhat", "redhat")
]

# ‘slavePortnum’ defines the TCP port to listen . This must match the value
# configured into the buildslaves (with their –master option)

c['slavePortnum'] = 9989

####### CHANGESOURCES

# the ‘change_source’ setting tells the buildmaster how it should find out
# about source code changes. Any class which implements IChangeSource can be
# put here: there are several in buildbot/changes/*.py to choose from.

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

####### SCHEDULERS

## configure the Schedulers

from buildbot import scheduler

stable = scheduler.Scheduler(name=”stable”, builderNames=["debian_stable", "freebsd_stable", "redhat_stable"], treeStableTimer=60, branch=”0.24.x”)
dev = scheduler.Scheduler(name=”dev”, builderNames=["debian_dev", "freebsd_dev", "redhat_dev"], treeStableTimer=60, branch=”master”)

c['schedulers'] = [stable, dev]

####### BUILDERS

# the ‘builders’ list defines the Builders. Each one configured with a
# dictionary, using the following keys:
#  name (required): the name used to describe this bilder
#  slavename (required): which slave to use, must appear in c['bots']
#  builddir (required): which subdirectory to run the builder in
#  factory (required): a BuildFactory to define how the build run
#  periodicBuildTime (optional): if set, force a build every N seconds

# buildbot/process/factory.py provides several BuildFactory classes you can
# start with, which implement build processes for common targets (GNU
# autoconf projects, CPAN perl modules, etc). The factory.BuildFactory the
# base class, and configured with a series of BuildSteps. When the build
# run, the appropriate buildslave told to execute each Step in turn.

# the first BuildStep typically responsible for obtaining a copy of the
# sources. There are source-obtaining Steps in buildbot/steps/source.py for
# CVS, , and others.

from buildbot.process import factory
from buildbot.steps import source, shell

pstable = factory.BuildFactory()
pstable.addStep(source.Git(repourl=’git://.com/jamtur01/puppet.git’, branch=’0.24.x’))
pstable.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pstable.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

pdev = factory.BuildFactory()
pdev.addStep(source.Git(repourl=’git://reductivelabs.com/puppet’, branch=’master’))
pdev.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pdev.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

debian_stable = {‘name’: “debian_stable”,
‘slavename’: “debian”,
‘builddir’: “debian_stable”,
‘factory’: pstable,
}

debian_dev = { ‘name’: “debian_dev”,
‘slavename’: “debian”,
‘builddir’: “debian_dev”,
‘factory’: pdev,
}

redhat_stable = {‘name’: “redhat_stable”,
‘slavename’: “redhat”,
‘builddir’: “redhat_stable”,
‘factory’: pstable,
}

redhat_dev = { ‘name’: “redhat_dev”,
‘slavename’: “redhat”,
‘builddir’: “redhat_dev”,
‘factory’: pdev,
}

freebsd_stable = {‘name’: “freebsd_stable”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_stable”,
‘factory’: pstable,
}

freebsd_dev = { ‘name’: “freebsd_dev”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_dev”,
‘factory’: pdev,
}

c['builders'] = [debian_stable, debian_dev, freebsd_stable, freebsd_dev, redhat_stable, redhat_dev]

####### STATUS TARGETS

# ‘status’ a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []

from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010))

from buildbot.status import mail
c['status'].append(mail.MailNotifier(fromaddr=”buildbot@reductivelabs.com”,
extraRecipients=["puppet-build@googlegroups.com"],
sendToInterestedUsers=False))

from buildbot.status import words
c['status'].append(words.IRC(host=”irc.freenode.net”, nick=”pinocchio”,
channels=["#puppet"],
password=”password”))

# from buildbot.status import client
# c['status'].append(client.PBListener(9988))

####### DEBUGGING OPTIONS

# if you set ‘debugPassword’, then you can connect to the buildmaster with
# the diagnostic tool in contrib/debugclient.py . From this tool, you can
# manually force builds and inject changes, which may be useful for testing
# your buildmaster without actually commiting changes to your (or
# before you have a functioning ‘sources’ set up). The debug tool uses the
# same port number as the slaves do: ‘slavePortnum’.

#c['debugPassword'] = “debugpassword”

# if you set ‘manhole’, you can ssh into the buildmaster and get an
# interactive python shell, which may be useful for debugging buildbot
# internals. It probably only useful for buildbot developers. You can also
# use an authorized_keys file, or plain telnet.
#from buildbot import manhole
#c['manhole'] = manhole.PasswordManhole(“tcp:9999:interface=127.0.0.1″,
#                                       “admin”, “password”)

####### PROJECT IDENTITY

# the ‘projectName’ string will be used to describe the project that this
# buildbot working . For example, it used as the title of the
# waterfall HTML page. The ‘projectURL’ string will be used to provide a link
# from buildbot HTML pages to your project’s home page.

c['projectName'] = “Puppet”
c['projectURL'] = “http://reductivelabs.com/trac/puppet/”

# the ‘buildbotURL’ string should point to the location where the buildbot’s
# internal web server (usually the html.Waterfall page) visible. This
# typically uses the port number set in the Waterfall ‘status’ entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.

#c['buildbotURL'] = “http://10.0.0.x:8010/”

3 comments »

Posted in buildbot, puppet

Tags:

Japanese journalist confirmed dead in Myanmar

September 28th, 2007

Our Man in Tokyo highlighted the death of journalist, Kenji Nagai, in Burma this week. Initial media reports suggested he was a victim of a stray bullet in the riots in Rangoon. Further information has emerged that suggests he was shot at point blank range whilst still attempting to photograph the demonstrations and the government’s crackdown the pro-democracy movement. It brings the number of journalists killed in 2007 to 35.

Growing up with a journalist as a father and now having one as a girlfriend I’ve always been incredibly angry about how journalists and writers are so frequently persecuted, tortured, imprisoned and murdered. Destruction of a free press, the silencing of opposing voices and hiding the truth are always one of the first steps taken by any regime or government that wants to oppress its people. Journalists, both foreign and domestic, in many countries have borne the brunt of this sort of persecution and danger. Too many journalists die unknown and unremembered or are disappeared as an inconvenience. Sadly, this trend seems to be getting worse and worse.

1 comment »

Posted in Uncategorized

Tags:

Stephen Colbert on John Howard and Barack Obama

February 13th, 2007

Just watched The Colbert Report as Stephen Colbert lambasted John Howard for his comments Barack Obama. So very, very funny. Apparently we Australians have balls the size of kookaburras but we should mind our own business and next time we’re waltzing matilda we should check out Matilda’s adam’s apple because dude she’s a MAN!

2 comments »

Posted in Uncategorized

Tags:

A solution to the Middle East Crisis

November 11th, 2006

(The result of some IMs with Ms Cynic the other night)

SCENE ONE: The United Nations.

Kofi: I move we should send the Priscilla Queen of the Desert Bus to Gaza.
Mamoud achmed hackenabachkk: I second the motion.

SCENE TWO: Gaza

Reporter: Katie, amazing scenes in Gaza. Jews and Muslims have united here today to gay bash visiting drag queens.
(Drag queen in heels and wig being stoned in the background – with onlooking group of cheering Muslims and Orthodox Jews)
Katie: Well Bob that’s a sight many of us never though we’d see, Jews and Muslims working together in common cause. Exciting times..

STATUS: Middle East situation resolved.

Also from same conversation – Brokeback Mountain 2: The Return to Soddom.

No comments »

Posted in Uncategorized

Tags:

Google Interview Goes Horribly Wrong… :)

November 6th, 2006

So I had an interview with Google in which I turned into a total gibbering idiot. Tongue-tied, lost for words, forgot simple things. I had worked myself up about it – read so many scary reports about it that I was a nervous wreck. Totally fluffed it – the guy was so nice – I was stammering at one point and he said “Look I know what you mean to say but you’re obviously a bit flustered – this what you meant?” And I said “SYN, SYN-ACK, ACK” about three times in a row and was unable to start the next sentence because I kept doing it. And then I forgot the basic data types in Perl and then I couldn’t remember what the definition of a ‘salt’ was – all I could think of was condiments. Which didn’t help.

Ah well. Interviews have always been my weakest point – I get stomach cramps, panic and my normally reasonably articulate self goes totally to shit. I did have a vague plan of having a nerve-steadying drink beforehand. I was all sensible and decided not to. Am now rather regretting that decision. At least I would have gone down with a higher-than-zero blood alcohol level…. :)

5 comments »

Posted in Blog

Tags:

Excellent script for automatically updating root hints file for Bind

May 20th, 2006

I was going to write a script to automatically update the root hints files my Bind servers. Then I found this excellent script – you will need to modify it to suit your own environment:

#!/bin/sh
#
# Update the nameserver cache information file once per month.
# This run automatically by a cron entry.
#
# Original by Al Longyear
# Updated for BIND 8 by Nicolai Langfeldt
# Miscelanious error-conditions reported by David A. Ranch
# Ping test suggested by Martin Foster
# named up-test suggested by Erik Bryer.
#
(
echo “To: root ”
echo “From: system ”

# named up? Check the status of named.
case `rndc status 2>&1` in
*refused*)
echo “named DOWN. root.hints was NOT updated”
echo
exit 0
;;
esac

PATH=/sbin:/usr/sbin:/bin:/usr/bin:
export PATH
# NOTE: /var/named/master must be writable only by trusted users or this script
# will cause root compromise/denial of service opportunities.
cd /var/named/master 2>/dev/null || {
echo “Subject: Cannot cd to /var/named/master, error $?”
echo
echo “The subject says it all”
exit 1
}

# Are we online? Ping a server at your ISP
case `ping -qnc 1 somehost.net 2>&1` in
*’100% packet loss’*)
echo “Subject: db.cache NOT updated. The network DOWN.”
echo
echo “The subject says it all”
exit 1
;;
esac

dig @e.root-servers.net . ns >db.cache.new 2> errors

case `cat db.cache.new` in
*NOERROR*)
# It worked
:;;
*)
echo “Subject: The db.cache file update has FAILED.”
echo
echo “The db.cache update has failed”
echo “This the dig output reported:”
echo
cat db.cache.new errors
exit 1
;;
esac

echo “Subject: The db.cache file has been updated”
echo
echo “The db.cache file has been updated to contain the following
information:”
echo
cat db.cache.new

chown named.named db.cache.new
chmod 444 db.cache.new
rm -f db.cache.old errors
mv db.cache db.cache.old
mv db.cache.new db.cache
rndc reload
echo
echo “The nameserver has been restarted to ensure that the update complete.”
echo “The previous root.hints file now called
/var/named/master/db.cache.old.”
) 2>&1 | /usr/lib/sendmail -t

exit 0

No comments »

Posted in Uncategorized

Tags:

Nagios book selling most well

May 10th, 2006

Absolutely thrilled – my new book Nagios 2.0 selling extremely well. Haven’t had any reviews yet but the consistently good Amazon sales suggest that it has been a hit. According to the publisher it has also done very well for technical books (via its’ Neilsen BookScan ratings). All in all it bloody great. According to Amazon 42% of the people who view the book’s page buy it.

I also poked into the Nagios stats for usage today which are housed the Nagios home page and realised a lot more people and organisations used it than I had previously realised. And that just the ones that reported their use. A lot of corporates also seem to use it without the desire to mention they are Nagios installed. Hopefully the book makes a good contribution to the body of knowledge about how to install and utilise Nagios. Very chuffed.

No comments »

Posted in Uncategorized

Tags: