Posts Tagged ‘repository’

Puppet Module Repository isn’t just for modules

June 1st, 2010

You can store more than just your modules at the Forge. :)   I just added my types and providers to my collection of modules at the new Puppet Module Forge.  I’d love to all those people maintaining types and providers, functions, and facts add theirs to the Forge also.  It’s a cool way to share your code (and the site allows you to provide links back to your code and ticketing system so user’s can report bugs).  In time I hope most people’s environments will consist of the core types and providers bundled with and a selection of cool code generated by the and sourced from the Forge.

No comments »

Posted in Blog, devops, open source, puppet

Tags:

Puppet Forge in beta!

May 27th, 2010

The Forge AKA the Puppet Module Repository is live and operational.  It’s a store of modules (and types and providers) that allows you to share your awesome code and modules with others.

It also comes with the -module tool that allows you to build modules for, manage and install modules from the forge.  You can install -module via a gem:

$ sudo gem install -module

Both the site and tool are in public beta right now so hammer away at it and tell us what you think!

1 comment »

Posted in Blog, devops, open source, puppet, rails

Tags:

Creating Puppet types and providers is easy…

February 1st, 2010

types are used to manage individual configuration items.  has a package , a service , a user , etc.  Each has providers. Each provider handles the management of that configuration on a different platform or tool, for example the package has aptitude, yum, RPM, and DMG providers (amongst 22 others – what is wrong with people that they need to invent new packaging systems… but I digress).

There are a lot of types, in fact I think covers a pretty good spectrum of configuration items that need to be managed.  I don’t know of anything in particular that is missing that I can’t live without.  But there are little gaps that are annoying, I’d like network and firewall types for example, but creating both these types in a generic enough way to support multiple platforms would be, IMHO, a non-trivial problem. 

Another gap is VCS/DVCS management. A lot of people use source code in repositories to do things with (including install stuff from you bad people – package things … it’s healthier). currently relies on creating and removing these repositories with the exec (which executes scripts or binaries), for example:

exec { "svn co http://core.svn.wordpress.org/trunk/ /var/www/wp":
    creates => "/var/www/wp",
}

This is a bit ugly and it’d be a lot easier to write a to manage repositories. But types and providers are written in Ruby and really, really complex and hard to develop. Right? Right?

No. No, they are not… and I’m going to create a simple and provider to show you. :)

Here’s a very (very!) simple , called repo, for managing repositories. I’ve created providers for SVN and Git as examples also. The first part of the repo is the itself – these are usually stored in lib// or distributed via modules (see the PluginsInModules page in the wiki). I’ll create a file called repo.rb.

$ touch repo.rb

And then populate the file:

Puppet::Type.newtype(:repo) do
    @doc = "Manage repos"
 
    ensurable
 
    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
 
    end
 
    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end
end

So – pretty simple. We create a block ::.newtype(:repo) do that creates a new , which we’ve called repo.

Inside the block we’ve got a @doc string. This is the documentation for the . Add whatever level of detail and examples in here that is required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. types use the ensure property to determine the state of a configuration item.

service { "sshd":
    ensure => present,
}

The ensurable statement tells to expect three methods: create, destroy and exists? in our provider. These methods, allow, respectively:

  • A command to create the resource
  • A command to delete the resource, and
  • A command to check for the existence of the resource

All we then need to do is specify these methods and their contents and creates the supporting infrastructure around them but more on this when we look at our providers.

Next, we’ve defined a new parameter – this one called source.

    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
    end

The source parameter will tell the repo where to go to retrieve/clone/checkout our source .

In this parameter we’re also using a hook called validate. Normally used to check the value for appropriateness here we’re using it to take a guess at what provider to use. Our code says, if the source parameter starts with git then use the Git provider, if not default to the Subversion provider. This is obviously fairly crude as a default and we can override this by defining the provider attribute in our resources:

provider => git,

We’ve also used another piece of automagic, isnamevar, to make this parameter the “name” variable for this . In -speak, the value of this parameter is used as the name of the resource.

(Types have two kinds of values – properties and parameters. Properties “do things”. They tell us HOW the provider works. We’ve only defined one property, ensure, by using the ensurable statement. Parameters are more like variables, they contain information relevant to configuring the resource the manages rather than “doing things”.)

Finally, we’ve defined another parameter, path.

    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end

This is a variable value that specifies where the repo should put the cloned/checked-out . In this parameter we’ve again used the validate hook to create a block that checks the value for appropriateness. Here we’re just checking, very crudely, to make sure it looks like the destination path is a valid fully-qualified file path. We could also use this validation for the source parameter to confirm a valid source URL/location was being provided.

(You can also use another hook called munge to adjust the value of the parameter rather than validating it before passing it to the provider.)

And that is it for the .

Next, we need to create a provider for our . Let’s start with a Subversion provider like so:

require 'fileutils'
 
Puppet::Type.type(:repo).provide(:svn) do
    desc "SVN Support"
 
    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"
 
    def create
        svncmd "checkout", resource[:name], resource[:path]
    end
 
    def destroy
        FileUtils.rm_rf resource[:path]
    end
 
    def exists?
        File.directory? resource[:path]
    end
end

Up front we’ve required the fileutils library, which we’re going to use a method from. Next, we’ve defined the provider as a block:

Puppet::Type.type(:repo).provide(:svn) do

We tell that this is a provider called svn for the called repo.

Then we use a desc method that allows us to add some documentation to our provider.

Next, we define the commands that this provider will use, here the svn and svnadmin binaries, to manipulate our resource’s configuration.

    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"

uses these commands to determine if the provider is appropriate to use on a client, if can’t find these commands in the local path then it will disable the provider.

Next, we’ve defined three methods – create, destroy and exists?. Sounds familiar? Yep, these are the methods that the ensurable statement expects to find in the provider:

The create method ensures our resource is created. It uses the svn command to create a with a source of resource[:name] (remember the source parameter in our is also the name variable of the – we could also specify resource[:source] here too) and a destination of resource[:path] (the value of the path attribute).

The delete method ensures the deletion of the resource. In this case, it deletes the directory and files specified in the resource[:path] parameter.

Lastly, the exists? method checks to see if the resource exists. Its operation is pretty simple and closely linked with the value of the ensure attribute in the resource:

  • If exists? is false and ensure is present, then create method will be called.
  • If exists? is true and ensure is set to absent, then the destroy method will be called.

In this case the exists? method checks if there is already a directory at the location specified in the resource[:path] parameter.

So, let’s put all this together and create a resource with our new . I’ve assumed you’ve already distributed your and providers to . We can then create a resource like:

repo { "wp":
    source => "http://core.svn.wordpress.org/trunk/",
    path => "/var/www/wp",
    ensure => present,
}

Simple eh? We specify a repo resource, the source we wish to check out or clone from, the destination path and the ensure attribute (present or absent) and that’s it.

You can see the complete code for this and its providers at my Puppet repository on GitHub. It’s obviously very basic but should be easy to extend to provide additional capabilities (and currently has no tests – my bad). You can find further documentation (in a lot more detail!) on creating your own types and providers at the Puppet wiki.

14 comments »

Posted in Blog, open source, puppet

Tags:

Puppet’s BuildBot

August 24th, 2008

So rather than doing the work I actually should be I’ve been playing with BuildBot. I had intended to get around to setting up BuildBot sometime in the next couple of months but I got hooked.

The reason I wanted to have a look at BuildBot was that Puppet has reached a stage where we simply can’t test every platform it runs on. We are also starting to get patches from a wider variety of sources. Buildbot will allow us to execute our tests on a wider variety of platforms. Hopefully with the cooperation of the we can gather a really big collection of build platforms to test on.

Here’s the blurb for BuildBot

The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.

The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin.

It’s a very easy tool to deploy. The hardest part has been the slightly broken Git source handling and the assumption that any Git is local. I need to have a local Git to allow BuildBot to submit the right commits references to the PBChangeSource function.

But I designed a basic process for handling new commits:

1. Commit pushed to .
2. Commit bot at picks up commit and sends it to BuildBot Master.
3. BuildBot uses the git_buildbot.py script to calculate the before/after commit and branch references and tell BuildBot about them.
4. BuildBot executes the build and tells each slave to retrieve the commit and runs the tests. Currently we’re running:

a. All the Unit tests
b. All the tests

5. We then get the results of the tests on the website and in an email to the new Builds mailing list.

In addition I’ve also enabled BuildBot’s IRC bot and added a new bot, called pinocchio, to the # channel that reports on build status.

At this stage it’s all in test mode and when I’ve ironed out a few issues we should be in a position to do a production installation at and start canvassing for build slaves.

UPDATE

After mucking around with Buildbot I just couldn’t get a whole bunch of issues with Git resolved so we changed to as our – which works much better.  The message overall is – and Git: still a young pair.  I’ve included our configuration below for edification:

# -*- python -*-
# ex: set syntax=python:

# This is a sample buildmaster config file. It must be installed as
# ‘master.cfg’ in your buildmaster’s base directory (although the filename
# can be changed with the –basedir option to ‘mktap buildbot master’).

# It has one job: define a dictionary named BuildmasterConfig. This
# dictionary has a variety of keys to control different aspects of the
# buildmaster. They are documented in docs/config.xhtml .

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### BUILDSLAVES

# the ‘slaves’ list defines the set of allowable buildslaves. Each element is
# a tuple of bot-name and bot-password. These correspond to values given to
# the buildslave’s mktap invocation.
from buildbot.buildslave import BuildSlave

c['slaves'] = [BuildSlave("debian", "debian"),
BuildSlave("freebsd", "freebsd"),
BuildSlave("redhat", "redhat")
]

# ‘slavePortnum’ defines the TCP port to listen on. This must match the value
# configured into the buildslaves (with their –master option)

c['slavePortnum'] = 9989

####### CHANGESOURCES

# the ‘change_source’ setting tells the buildmaster how it should find out
# about source code changes. Any class which implements IChangeSource can be
# put here: there are several in buildbot/changes/*.py to choose from.

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

####### SCHEDULERS

## configure the Schedulers

from buildbot import scheduler

stable = scheduler.Scheduler(name=”stable”, builderNames=["debian_stable", "freebsd_stable", "redhat_stable"], treeStableTimer=60, branch=”0.24.x”)
dev = scheduler.Scheduler(name=”dev”, builderNames=["debian_dev", "freebsd_dev", "redhat_dev"], treeStableTimer=60, branch=”master”)

c['schedulers'] = [stable, dev]

####### BUILDERS

# the ‘builders’ list defines the Builders. Each one is configured with a
# dictionary, using the following keys:
#  name (required): the name used to describe this bilder
#  slavename (required): which slave to use, must appear in c['bots']
#  builddir (required): which subdirectory to run the builder in
#  factory (required): a BuildFactory to define how the build is run
#  periodicBuildTime (optional): if set, force a build every N seconds

# buildbot/process/factory.py provides several BuildFactory classes you can
# start with, which implement build processes for common targets (GNU
# autoconf projects, CPAN perl modules, etc). The factory.BuildFactory is the
# base class, and is configured with a series of BuildSteps. When the build
# is run, the appropriate buildslave is told to execute each Step in turn.

# the first BuildStep is typically responsible for obtaining a copy of the
# sources. There are source-obtaining Steps in buildbot/steps/source.py for
# CVS, SVN, and others.

from buildbot.process import factory
from buildbot.steps import source, shell

pstable = factory.BuildFactory()
pstable.addStep(source.Git(repourl=’git://.com/jamtur01/.git’, branch=’0.24.x’))
pstable.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pstable.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

pdev = factory.BuildFactory()
pdev.addStep(source.Git(repourl=’git://.com/’, branch=’master’))
pdev.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pdev.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

debian_stable = {‘name’: “debian_stable”,
‘slavename’: “debian”,
‘builddir’: “debian_stable”,
‘factory’: pstable,
}

debian_dev = { ‘name’: “debian_dev”,
‘slavename’: “debian”,
‘builddir’: “debian_dev”,
‘factory’: pdev,
}

redhat_stable = {‘name’: “redhat_stable”,
‘slavename’: “redhat”,
‘builddir’: “redhat_stable”,
‘factory’: pstable,
}

redhat_dev = { ‘name’: “redhat_dev”,
‘slavename’: “redhat”,
‘builddir’: “redhat_dev”,
‘factory’: pdev,
}

freebsd_stable = {‘name’: “freebsd_stable”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_stable”,
‘factory’: pstable,
}

freebsd_dev = { ‘name’: “freebsd_dev”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_dev”,
‘factory’: pdev,
}

c['builders'] = [debian_stable, debian_dev, freebsd_stable, freebsd_dev, redhat_stable, redhat_dev]

####### STATUS TARGETS

# ‘status’ is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []

from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010))

from buildbot.status import mail
c['status'].append(mail.MailNotifier(fromaddr=”buildbot@.com”,
extraRecipients=["-build@googlegroups.com"],
sendToInterestedUsers=False))

from buildbot.status import words
c['status'].append(words.IRC(host=”irc.freenode.net”, nick=”pinocchio”,
channels=["#"],
password=”password”))

# from buildbot.status import client
# c['status'].append(client.PBListener(9988))

####### DEBUGGING OPTIONS

# if you set ‘debugPassword’, then you can connect to the buildmaster with
# the diagnostic tool in contrib/debugclient.py . From this tool, you can
# manually force builds and inject changes, which may be useful for testing
# your buildmaster without actually commiting changes to your (or
# before you have a functioning ‘sources’ set up). The debug tool uses the
# same port number as the slaves do: ‘slavePortnum’.

#c['debugPassword'] = “debugpassword”

# if you set ‘manhole’, you can ssh into the buildmaster and get an
# interactive python shell, which may be useful for debugging buildbot
# internals. It is probably only useful for buildbot developers. You can also
# use an authorized_keys file, or plain telnet.
#from buildbot import manhole
#c['manhole'] = manhole.PasswordManhole(“tcp:9999:interface=127.0.0.1″,
#                                       “admin”, “password”)

####### PROJECT IDENTITY

# the ‘projectName’ string will be used to describe the project that this
# buildbot is working on. For example, it is used as the title of the
# waterfall HTML page. The ‘projectURL’ string will be used to provide a link
# from buildbot HTML pages to your project’s home page.

c['projectName'] = “”
c['projectURL'] = “http://.com/trac//”

# the ‘buildbotURL’ string should point to the location where the buildbot’s
# internal web server (usually the html.Waterfall page) is visible. This
# typically uses the port number set in the Waterfall ‘status’ entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.

#c['buildbotURL'] = “http://10.0.0.x:8010/”

3 comments »

Posted in buildbot, puppet

Tags: