So rather than doing the work I actually should be I’ve been playing with BuildBot. I had intended to get around to setting up BuildBot sometime in the next couple of months but I got hooked.

The reason I wanted to have a look at BuildBot was that Puppet has reached a stage where we simply can’t test every platform it runs on. We are also starting to get patches from a wider variety of sources. Buildbot will allow us to execute our tests on a wider variety of platforms. Hopefully with the cooperation of the community we can gather a really big collection of build platforms to test on.

Here’s the blurb for BuildBot

The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.

The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin.

It’s a very easy tool to deploy. The hardest part has been the slightly broken Git source handling and the assumption that any Git repository is local. I need to have a local Git repository to allow BuildBot to submit the right commits references to the PBChangeSource function.

But I designed a basic process for handling new commits:

1. Commit pushed to GitHub.
2. Commit bot at GitHub picks up commit and sends it to BuildBot Master.
3. BuildBot uses the git_buildbot.py script to calculate the before/after commit and branch references and tell BuildBot about them.
4. BuildBot executes the build and tells each slave to retrieve the commit and runs the tests. Currently we’re running:

a. All the Unit tests
b. All the RSpec tests

5. We then get the results of the tests on the website and in an email to the new Puppet Builds mailing list.

In addition I’ve also enabled BuildBot’s IRC bot and added a new bot, called pinocchio, to the #puppet channel that reports on build status.

At this stage it’s all in test mode and when I’ve ironed out a few issues we should be in a position to do a production installation at ReductiveLabs and start canvassing for build slaves.

UPDATE

After mucking around with Buildbot I just couldn’t get a whole bunch of issues with Git resolved so we changed to Hudson as our CI – which works much better.  The message overall is – CI and Git: still a young pair.  I’ve included our configuration below for edification:

# -*- python -*-
# ex: set syntax=python:

# This is a sample buildmaster config file. It must be installed as
# ‘master.cfg’ in your buildmaster’s base directory (although the filename
# can be changed with the –basedir option to ‘mktap buildbot master’).

# It has one job: define a dictionary named BuildmasterConfig. This
# dictionary has a variety of keys to control different aspects of the
# buildmaster. They are documented in docs/config.xhtml .

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### BUILDSLAVES

# the ‘slaves’ list defines the set of allowable buildslaves. Each element is
# a tuple of bot-name and bot-password. These correspond to values given to
# the buildslave’s mktap invocation.
from buildbot.buildslave import BuildSlave

c['slaves'] = [BuildSlave("debian", "debian"),
BuildSlave("freebsd", "freebsd"),
BuildSlave("redhat", "redhat")
]

# ‘slavePortnum’ defines the TCP port to listen on. This must match the value
# configured into the buildslaves (with their –master option)

c['slavePortnum'] = 9989

####### CHANGESOURCES

# the ‘change_source’ setting tells the buildmaster how it should find out
# about source code changes. Any class which implements IChangeSource can be
# put here: there are several in buildbot/changes/*.py to choose from.

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

####### SCHEDULERS

## configure the Schedulers

from buildbot import scheduler

stable = scheduler.Scheduler(name=”stable”, builderNames=["debian_stable", "freebsd_stable", "redhat_stable"], treeStableTimer=60, branch=”0.24.x”)
dev = scheduler.Scheduler(name=”dev”, builderNames=["debian_dev", "freebsd_dev", "redhat_dev"], treeStableTimer=60, branch=”master”)

c['schedulers'] = [stable, dev]

####### BUILDERS

# the ‘builders’ list defines the Builders. Each one is configured with a
# dictionary, using the following keys:
#  name (required): the name used to describe this bilder
#  slavename (required): which slave to use, must appear in c['bots']
#  builddir (required): which subdirectory to run the builder in
#  factory (required): a BuildFactory to define how the build is run
#  periodicBuildTime (optional): if set, force a build every N seconds

# buildbot/process/factory.py provides several BuildFactory classes you can
# start with, which implement build processes for common targets (GNU
# autoconf projects, CPAN perl modules, etc). The factory.BuildFactory is the
# base class, and is configured with a series of BuildSteps. When the build
# is run, the appropriate buildslave is told to execute each Step in turn.

# the first BuildStep is typically responsible for obtaining a copy of the
# sources. There are source-obtaining Steps in buildbot/steps/source.py for
# CVS, SVN, and others.

from buildbot.process import factory
from buildbot.steps import source, shell

pstable = factory.BuildFactory()
pstable.addStep(source.Git(repourl=’git://github.com/jamtur01/puppet.git’, branch=’0.24.x’))
pstable.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pstable.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

pdev = factory.BuildFactory()
pdev.addStep(source.Git(repourl=’git://reductivelabs.com/puppet’, branch=’master’))
pdev.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pdev.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

debian_stable = {‘name’: “debian_stable”,
‘slavename’: “debian”,
‘builddir’: “debian_stable”,
‘factory’: pstable,
}

debian_dev = { ‘name’: “debian_dev”,
‘slavename’: “debian”,
‘builddir’: “debian_dev”,
‘factory’: pdev,
}

redhat_stable = {‘name’: “redhat_stable”,
‘slavename’: “redhat”,
‘builddir’: “redhat_stable”,
‘factory’: pstable,
}

redhat_dev = { ‘name’: “redhat_dev”,
‘slavename’: “redhat”,
‘builddir’: “redhat_dev”,
‘factory’: pdev,
}

freebsd_stable = {‘name’: “freebsd_stable”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_stable”,
‘factory’: pstable,
}

freebsd_dev = { ‘name’: “freebsd_dev”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_dev”,
‘factory’: pdev,
}

c['builders'] = [debian_stable, debian_dev, freebsd_stable, freebsd_dev, redhat_stable, redhat_dev]

####### STATUS TARGETS

# ‘status’ is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []

from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010))

from buildbot.status import mail
c['status'].append(mail.MailNotifier(fromaddr=”buildbot@reductivelabs.com”,
extraRecipients=["puppet-build@googlegroups.com"],
sendToInterestedUsers=False))

from buildbot.status import words
c['status'].append(words.IRC(host=”irc.freenode.net”, nick=”pinocchio”,
channels=["#puppet"],
password=”password”))

# from buildbot.status import client
# c['status'].append(client.PBListener(9988))

####### DEBUGGING OPTIONS

# if you set ‘debugPassword’, then you can connect to the buildmaster with
# the diagnostic tool in contrib/debugclient.py . From this tool, you can
# manually force builds and inject changes, which may be useful for testing
# your buildmaster without actually commiting changes to your repository (or
# before you have a functioning ‘sources’ set up). The debug tool uses the
# same port number as the slaves do: ‘slavePortnum’.

#c['debugPassword'] = “debugpassword”

# if you set ‘manhole’, you can ssh into the buildmaster and get an
# interactive python shell, which may be useful for debugging buildbot
# internals. It is probably only useful for buildbot developers. You can also
# use an authorized_keys file, or plain telnet.
#from buildbot import manhole
#c['manhole'] = manhole.PasswordManhole(“tcp:9999:interface=127.0.0.1″,
#                                       “admin”, “password”)

####### PROJECT IDENTITY

# the ‘projectName’ string will be used to describe the project that this
# buildbot is working on. For example, it is used as the title of the
# waterfall HTML page. The ‘projectURL’ string will be used to provide a link
# from buildbot HTML pages to your project’s home page.

c['projectName'] = “Puppet”
c['projectURL'] = “http://reductivelabs.com/trac/puppet/”

# the ‘buildbotURL’ string should point to the location where the buildbot’s
# internal web server (usually the html.Waterfall page) is visible. This
# typically uses the port number set in the Waterfall ‘status’ entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.

#c['buildbotURL'] = “http://10.0.0.x:8010/”