Archive for the ‘Blog’ category

What DevOps means to me…

February 19th, 2010

Over the last year or so a bunch of presumptuous European sysadmins and developers, joined by some of their American brethren and even a couple of us antipodeans (there are others too!) have been talking about a concept called DevOps.  DevOps is the merger of the realms of development and operations (and if truth be told elements of product management,QA, and *winces* even sales should be thrown into the mix too).

The Broken

So … why should we merge or bring together the two realms?  Well there are lots of reasons but first and foremost because what we’re doing now is broken.  Really, really broken.  In many shops the relationship between development (or engineering) and operations is dysfunctional to the point of occasional toxicity.

Here’s an example I think everyone will be at least partially familiar with: the minefield that is project to production software deployment.  Curse along as I explain.

Development builds an application, the new hotness which promises customers all the whizz-bang features and will make the company millions.  It is built using cutting edge technology and a brand new platform and it has got to be delivered right now.  Development cuts code like crazy and gets the product ready for market ahead of schedule.  They throw their masterpiece over the fence to Operations to implement and dash off to the pub for the wrap party.

Operations catches the deployment and is filled with horror.

The Operations team summarises their horror and says one or more of:

  • The wonder application won’t run on our infrastructure because {it’s too old, it doesn’t have capacity, we don’t support that version}
  • The architecture of the application doesn’t match our { storage, network, deployment, security } model
  • We weren’t consulted about the { reporting, security, monitoring, backup, provisioning } and it can’t be “productionised”.

But Operations persevere and install the new hotness – cursing and bitching throughout.  Sadly, after forcing the application onto infrastructure and bending and twisting the architecture to get it running, the performance of the new application can be summed up as “epic fail”.

Operations sighs and starts logging problems and passing issues back to the Development team.  Their responses generally come from the following pool:

  • It’s not our fault – our code is perfect – it’s just been poorly implemented
  • Operations are stupid and don’t understand the new hotness – why can’t they implement the cutting edge technology? Why are they so backward?
  • It runs fine on my machine…

The interactions between teams quickly becomes a toxic blame storm. The customers (and by extension the shareholders, investors and management) then become the losers.  The loop gets closed with the company losing bucket loads of money and everyone losing their jobs.  EPIC and FAIL.

What’s different about DevOps?

DevOps is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups. In a DevOps environment, developers and sysadmins build relationships, processes, and tools that allow them to better interact and ultimately better service the customer.

DevOps is also more than just software deployment – it’s a whole new way of thinking about cooperation and coordination between the people who make the software and the people who run it.  Areas like automation, monitoring, capacity planning & performance, backup & recovery, security, networking and provisioning can all benefit from using a DevOps model to enhance the nature and quality of interactions between development and operations teams.

Everyone in the DevOps community has a slightly different take on “What is DevOps?”  We all bring different experiences and focuses to the problem space.  I personally see DevOps as having four quadrants:

Simplicity

KISS is King and in that vein this section is simple too. Design simple, repeatable, and reusable solutions. Simplicity saves documentation, training, and support time.  Simplicity increases the speed of communication, avoids confusion, and helps reduces the risk of development and operational errors.  Simplicity gets you to the pub faster.

Relationships

Engage early, engage often. Development teams need to embed operations people into their project and development life cycles.  Invite operational people to your scrum or development meetings.  Share ideas and information about product plans and new technologies. Gather operational requirements when gathering functional ones. As a project progresses test deployment, backup, monitoring, security and configuration management as well as application functionality.  The more issues you fix during the project the less issues you expose your customers to when the application is live.  Educate operations people about the applications architecture and the code base. The more information operations people can feed you about a problem with the code the less trouble-shooting you need to perform and the faster the problem can be fixed.

Operations people need to bring development people into the problem and change management space. Invite developers into your team meetings. Share your roadmaps and upgrade plans.  Understand where future development is heading to better ensure infrastructure deployments match product requirements.  Developers also bring skills, knowledge and tools that can help make your environment easier to manage, more efficient and cleaner. Learn to code or if you’re a hack-n-slash systems programmer like me then learn to code better. Concepts like building tools with APIs rather than closed interfaces, distributed version control, test driven development, and methodologies like Agile Development, Kanban and Scrum can revolutionise operational practises in the same way they’ve changed the way code is cut.

Don’t be afraid of ideas and approaches from outside your domain – we can all learn things, even if it’s “let’s never do it that way again…!”, from how others do things and ultimately? Guess what? Yep, we’re all on the SAME team.

Remember that interactions between people rank, in decreasing order of effectiveness (in IMHO but backed by some research):

  1. Face to face
  2. Video conference
  3. Phone
  4. IM & IRC
  5. Email

Process

Don’t underestimate the power of process and automation.  Many shops do process engineering – ranging from hand-written lists to ISO9001. Those processes generally have one key flaw: they focus on the outcome and its inevitability.  A simple process might provision a host – Step 1 install machine, Step 2 cable machine, Step 3 install OS, etc, etc. Assuming all goes to process then at the end of Step x you will have a fully provisioned host. But what happens if it doesn’t go right?  If your process breaks or you receive some anomalous output how does your process deal with it?  Instead think about process as a journey and map out the potential pitfalls and obstacles.  Treat your processes like applications and build error handling into them.  You can’t predict every application or operational pitfall or issue but you can ensure that if you hit one your process isn’t derailed.

Link process together across domains – software deployment, monitoring, capacity planning and other “operational” processes have their start in the development world.  Software deployment is the logical conclusion of the software development life cycle and should be viewed as such rather than a separate operational process. Another example is metrics and monitoring, it is hard to measure anything  without understanding the baselines and assumptions made in the development domain.  Joint processes also mean more opportunity for development and operations interaction, understanding and joint accountability. Finally, joint process development means single repositories for documentation and other opportunities for economies of scale.

Automate, automate, automate. Build or make use of simple and extensible tools (make sure they have APIs and machine readable input and output – see James White’s Infrastructure Manifesto).  Use tools like Puppet (or others) to manage your configuration.  Remember to extend your automation umbrella cross-domain and end-to-end in your environment – manage development, testing, staging and production environments with the same tools and processes.  Not only does this have economies of scale benefits in support and management but it means you can test deployment and management alongside functionality as your application and new codes rolls toward production.

Finally, when building process and automation always keep the KISS principle in mind. Complexity breeds opportunities for error. Build simple processes and tools that are easy to implement, manage and maintain.

Continuous Improvement

Don’t stop innovating and learning.  Technology moves fast.  So do customer requirements. Build continuous improvement and integration into your tools and processes.  Here is a good place operations people can learn from (good) developers about practises like test-driven development.  A good example here is to build tests for your software deployment process and infrastructure.  They are often an application in their own right and should be developed and maintained correctly. Your monitoring could also be extended with behavioural testing to deliver better business value.  Look at using development domain tools, like Hudson for example, to explore and measure the operational domain.

Learn from mistakes and from outages.  Seek root cause aggressively AND cross-domain.  If you have an outage and a post-incident review then bring development and operational teams together to review the incident.  Sometimes some simple code refactoring can save making infrastructure changes.  Work together to fix root cause, treat it with the same process you develop to conduct project to production software deployment, rather than relegating them to incident review reports or batting issues between teams.

Me

Finally, for me DevOps is about people and nature of the environment you want to work in.  The best thing about the movement for me is that it is trying to foster behaviours and environments where people work together towards joint goals rather than at cross-purposes or at odds.  That’s a world I’d much rather use my skills in.

5 comments »

Posted in Blog, devops, puppet

Tags:

Puppet ParsedFile types and providers

February 13th, 2010

In a recent post I talked about how easy it is to generate Puppet types and providers. In that post I used the example of a very simple Subversion and Git repository type, called repo. I’d like to show another example of a type and provider, this one used to manage the contents of the /etc/shells file. This type and provider makes use of some built-in Puppet functionality that allows the simple parsing of files and the management of their contents. To do this Puppet has a provider called ParsedFile that can be included into your own providers to provide this functionality.

Let’s start with our type:

Puppet::Type.newtype(:shells) do
    @doc = "Manage the contents of /etc/shells
 
            shells { "/bin/newshell":
                ensure => present,
            }"
 
    ensurable
 
    newparam(:shell) do
        desc "The shell to manage"
 
        isnamevar
 
    end
 
    newproperty(:target) do
        desc "Location of the shells file"
 
        defaultto {
            if
                @resource.class.defaultprovider.ancestors.include? (Puppet::Provider::ParsedFile)
                @resource.class.defaultprovider.default_target
            else
                nil
            end
        }
    end
end

So – pretty simple. We create a block Puppet::Type.newtype(:shells) do that creates a new type, which we’ve called shells. Inside the block we’ve got a @doc string. This is the documentation for the type. Add whatever level of detail and examples in here that is required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. Puppet types use the ensure property to determine the state of a configuration item. In our previous example, ensurable resulted in three methods in the provider: create, destroy, and exists?. In a ParsedFile provider we don’t use these methods at all as we’ll see shortly but rather specify how to handle each record in the file.

We’ve defined a new parameter – this one called shell.

newparam(:shell) do
        desc "The shell to manage"
 
        isnamevar
end

The shell parameter is the shell we’re going to manage in the /etc/shells file. We’ve also used another piece of Puppet automagic, isnamevar, to make this parameter the “name” variable for this type. In Puppet-speak, the value of this parameter is used as the name of the resource.

Lastly in our type we’ve specified an optional parameter, target, that allows us to override the default location of the shells file, usually /etc/shells.

newproperty(:target) do
        desc "Location of the shells file"
 
        defaultto {
            if
                @resource.class.defaultprovider.ancestors.include? (Puppet::Provider::ParsedFile)
                @resource.class.defaultprovider.default_target
            else
                nil
            end
        }
end

The target parameter is optional and would only be specified if the shells file wasn’t located in the /etc/ directory. It uses the defaultto structure to specify that the default value for the parameter is the value of default_target variable in the provider.

The provider for our type is also very simple:

require 'puppet/provider/parsedfile'
shells = "/etc/shells"
 
Puppet::Type.type(:shells).provide(:parsed, :parent => Puppet::Provider::ParsedFile, :default_target => shells, :filetype => :flat) do
 
    desc "The shells provider that uses the ParsedFile class"
 
    text_line :comment, :match => /^#/;
    text_line :blank, :match => /^\s*$/;
 
    record_line :parsed, :fields => %w{name}
end

The shells provider is stored in a file called parsed.rb in a directory named for the provider in the provider directory, for example:

/usr/lib/ruby/site_ruby/1.8/puppet/type/shells.rb
/usr/lib/ruby/site_ruby/1.8/puppet/provider/shells/parsed.rb

The file needs to be named parsed.rb to allow Puppet to load the ParsedFile support.

We first include the ParsedFile provider code at the top of our provider, require 'puppet/provider/parsedfile' and set a variable called shells to the location of the /etc/shells file. We’re going to use this variable a bit later.

Then we tell Puppet that this is a provider called shells. We specify a :parent value that tells Puppet that this provider should inherit the ParsedFile provider and make its functions available. We then specify the :default_target value to the shells variable we’ve just created. This tells the provider, that unless it is overridden by the target attribute, that the file to act upon is /etc/shells.

Then we use a desc method that allows us to add some documentation to our provider.

The next lines are the core of the provider. They tell the Puppet how to manipulate the target file to add or remove the required shell. The first two lines, both text_lines, tell Puppet how to match comments and blank lines respectively.

    text_line :comment, :match => /^#/;
    text_line :blank, :match => /^\s*$/;

We specify these to let Puppet know to ignore these lines as unimportant.

The next line performs the actual parsing of the relevant line in the /etc/shells file:

    record_line :parsed, :fields => %w{name}

The record_line parses each line and divides it into fields, in our case we only have one field: name. The name in this case is the shell we want to manage. So if we specify:

shells { "/bin/newshell":
    ensure => present,

Then Puppet would use the provider to add the /bin/newshell by parsing each line of the /etc/shells file and checking if the newshell is present. If it is, then Puppet will do nothing. If not, then Puppet will add newshell to the file. If we changed the ensure attribute to absent then Puppet would go through the file and remove the newshell if it is present.

It is important to remember that ParsedFile providers do have some limitations, they aren’t good at managing complex files such as configuration files with multi-line options, they are best for simple files that contain single line lists of entries such as the cron file entries or the /etc/hosts and /etc/shells files.

You can see the complete code for this type and its providers at my Puppet repository on GitHub. Quite a lot of the existing Puppet types and providers use ParsedFile providers (the cron type for example) and you can use these as examples of how to create your own providers. You can also find further documentation (in a lot more detail!) on creating your own types and providers at the Puppet wiki.

No comments »

Posted in Blog, open source, puppet

Tags:

Help Vampires: A Spotter’s Guide

February 9th, 2010

This is an excellent post on the phenomena of “help vampires” who suck up the support goodwill in communities.  It tells you how to spot them, how to handle them and how to protect your community against them.

No comments »

Posted in Blog, open source

Tags:

Creating Puppet types and providers is easy…

February 1st, 2010

Puppet types are used to manage individual configuration items.  Puppet has a package type, a service type, a user type, etc.  Each type has providers. Each provider handles the management of that configuration on a different platform or tool, for example the package type has aptitude, yum, RPM, and DMG providers (amongst 22 others – what is wrong with people that they need to invent new packaging systems… but I digress).

There are a lot of types, in fact I think Puppet covers a pretty good spectrum of configuration items that need to be managed.  I don’t know of anything in particular that is missing that I can’t live without.  But there are little gaps that are annoying, I’d like network and firewall types for example, but creating both these types in a generic enough way to support multiple platforms would be, IMHO, a non-trivial problem. 

Another gap is VCS/DVCS management. A lot of people use source code in repositories to do things with (including install stuff from you bad people – package things … it’s healthier). Puppet currently relies on creating and removing these repositories with the exec type (which executes scripts or binaries), for example:

exec { "svn co http://core.svn.wordpress.org/trunk/ /var/www/wp":
    creates => "/var/www/wp",
}

This is a bit ugly and it’d be a lot easier to write a Puppet type to manage repositories. But Puppet types and providers are written in Ruby and really, really complex and hard to develop. Right? Right?

No. No, they are not… and I’m going to create a simple type and provider to show you. :)

Here’s a very (very!) simple Puppet type, called repo, for managing repositories. I’ve created providers for SVN and Git as examples also. The first part of the repo type is the type itself – these are usually stored in lib/puppet/type or distributed via modules (see the PluginsInModules page in the Puppet wiki). I’ll create a file called repo.rb.

$ touch repo.rb

And then populate the file:

Puppet::Type.newtype(:repo) do
    @doc = "Manage repos"
 
    ensurable
 
    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
 
    end
 
    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end
end

So – pretty simple. We create a block Puppet::Type.newtype(:repo) do that creates a new type, which we’ve called repo.

Inside the block we’ve got a @doc string. This is the documentation for the type. Add whatever level of detail and examples in here that is required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. Puppet types use the ensure property to determine the state of a configuration item.

service { "sshd":
    ensure => present,
}

The ensurable statement tells Puppet to expect three methods: create, destroy and exists? in our provider. These methods, allow, respectively:

  • A command to create the resource
  • A command to delete the resource, and
  • A command to check for the existence of the resource

All we then need to do is specify these methods and their contents and Puppet creates the supporting infrastructure around them but more on this when we look at our providers.

Next, we’ve defined a new parameter – this one called source.

    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
    end

The source parameter will tell the repo type where to go to retrieve/clone/checkout our source repository.

In this parameter we’re also using a hook called validate. Normally used to check the value for appropriateness here we’re using it to take a guess at what provider to use. Our code says, if the source parameter starts with git then use the Git provider, if not default to the Subversion provider. This is obviously fairly crude as a default and we can override this by defining the provider attribute in our resources:

provider => git,

We’ve also used another piece of Puppet automagic, isnamevar, to make this parameter the “name” variable for this type. In Puppet-speak, the value of this parameter is used as the name of the resource.

(Types have two kinds of values – properties and parameters. Properties “do things”. They tell us HOW the provider works. We’ve only defined one property, ensure, by using the ensurable statement. Parameters are more like variables, they contain information relevant to configuring the resource the type manages rather than “doing things”.)

Finally, we’ve defined another parameter, path.

    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end

This is a variable value that specifies where the repo type should put the cloned/checked-out repository. In this parameter we’ve again used the validate hook to create a block that checks the value for appropriateness. Here we’re just checking, very crudely, to make sure it looks like the destination path is a valid fully-qualified file path. We could also use this validation for the source parameter to confirm a valid source URL/location was being provided.

(You can also use another hook called munge to adjust the value of the parameter rather than validating it before passing it to the provider.)

And that is it for the type.

Next, we need to create a provider for our type. Let’s start with a Subversion provider like so:

require 'fileutils'
 
Puppet::Type.type(:repo).provide(:svn) do
    desc "SVN Support"
 
    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"
 
    def create
        svncmd "checkout", resource[:name], resource[:path]
    end
 
    def destroy
        FileUtils.rm_rf resource[:path]
    end
 
    def exists?
        File.directory? resource[:path]
    end
end

Up front we’ve required the fileutils library, which we’re going to use a method from. Next, we’ve defined the provider as a block:

Puppet::Type.type(:repo).provide(:svn) do

We tell Puppet that this is a provider called svn for the type called repo.

Then we use a desc method that allows us to add some documentation to our provider.

Next, we define the commands that this provider will use, here the svn and svnadmin binaries, to manipulate our resource’s configuration.

    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"

Puppet uses these commands to determine if the provider is appropriate to use on a client, if Puppet can’t find these commands in the local path then it will disable the provider.

Next, we’ve defined three methods – create, destroy and exists?. Sounds familiar? Yep, these are the methods that the ensurable statement expects to find in the provider:

The create method ensures our resource is created. It uses the svn command to create a repository with a source of resource[:name] (remember the source parameter in our type is also the name variable of the type – we could also specify resource[:source] here too) and a destination of resource[:path] (the value of the path attribute).

The delete method ensures the deletion of the resource. In this case, it deletes the directory and files specified in the resource[:path] parameter.

Lastly, the exists? method checks to see if the resource exists. Its operation is pretty simple and closely linked with the value of the ensure attribute in the resource:

  • If exists? is false and ensure is present, then create method will be called.
  • If exists? is true and ensure is set to absent, then the destroy method will be called.

In this case the exists? method checks if there is already a directory at the location specified in the resource[:path] parameter.

So, let’s put all this together and create a resource with our new type. I’ve assumed you’ve already distributed your type and providers to Puppet. We can then create a resource like:

repo { "wp":
    source => "http://core.svn.wordpress.org/trunk/",
    path => "/var/www/wp",
    ensure => present,
}

Simple eh? We specify a repo resource, the source we wish to check out or clone from, the destination path and the ensure attribute (present or absent) and that’s it.

You can see the complete code for this type and its providers at my Puppet repository on GitHub. It’s obviously very basic but should be easy to extend to provide additional capabilities (and currently has no tests – my bad). You can find further documentation (in a lot more detail!) on creating your own types and providers at the Puppet wiki.

9 comments »

Posted in Blog, open source, puppet

Tags:

Puppet 0.25.4 released!

January 29th, 2010

You wanted “release early, release often” and the Puppet team has delivered!
The 0.25.4 release is a maintenance release (with one important feature – pre/post transaction hooks – discussed below) in the 0.25.x branch.  The release primarily addresses a regression introduced in 0.25.3 that caused issues with creating cron jobs.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.4.tar.gz

http://reductivelabs.com/downloads/gems/puppet-0.25.4.gem

http://gemcutter.org/gems/puppet

Please note that all final releases of Puppet are signed with the Reductive Labs key – http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying…

Please report feedback via the Reductive Labs Redmine site: http://projects.reductivelabs.com

Please select an affected version of 0.25.4

RELEASE NOTES

Pre/Post Transaction hooks

There is a new feature in this release: pre and post transaction hooks.  These hooks allow you to specify commands that should be run pre and post a Puppet configuration transaction.   They are set with the prerun_command and postrun_command settings in the puppet.conf configuration file.

prerun_command = /bin/runbeforetransaction
postrun_command = /bin/runaftertransaction

The command must exit with 0, i.e. succeed, otherwise the transaction will fail – if the pre command fails before the transaction is run and if the post command fails at the end of the transaction.

CHANGELOG
*  Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #3001: Can’t manage broken links
* Bug #3039: 0.25.3 gem spec specifies the executables incorrectly
* Bug #3075: sshkey host aliases broken by fix for #2813
* Bug #3088: Puppetd fails to stop after receiving SIGTERM
* Bug #3089: puppetlast gsub! error
* Bug #3093: Blastwave provider broken in 0.25.3
* Bug #3104: Test failed: Puppet::Network::XMLRPCClient when performing the rpc call and an exception is  raised.should log and raise XMLRPCClientError if Timeout::Error is raised
* Bug #3112: Problem with adding and removing crons
* Bug #3122: Uncharacterized failure in fileserving under OS X
* Bug #3125: Dpkg tests failing
* Feature #2914: Transactions should have before and after hooks

No comments »

Posted in Blog, open source, puppet

Tags:

The Tortoise and not the Hare 2 – Principles

January 24th, 2010

Kanban

In my first post I introduced you to the Toyota Production System and the Kanban signalling system. At the core of the TPS is the concept of maintaining efficiency and eliminating waste.  To govern this processes the TPS has a series of basic principles that articulate how this is achieved:

  1. Create continuous process flow to bring problems to the surface
  2. Use the “pull” system to avoid overproduction
  3. Level out or smooth the workload AKA “Heijunka” – be the tortoise not the hare
  4. Build a culture of stopping to fix problems, to get quality right from the first
  5. Standardized tasks are the foundation for continuous improvement and employee empowerment
  6. Use visual control so no problems are hidden
  7. Use only reliable, thoroughly tested technology that serves your people and processes.

I’ve skipped all the TPS rules around long-term thinking (for example the first principle of the TPS – Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals), corporate harmony, people development and organisational learning.  I haven’t skipped them because they aren’t important but because they aren’t immediately relevant to this discussion.  If you haven’t got the others right too though I suspect your organisation will go pear-shaped in other ways.

In each of the subsequent posts I am going to look at one of the seven principles I’ve articulated above, starting with exploring how you can make continuous process flow work for you.

No comments »

Posted in Blog, devops

Tags:

Puppet, Chef, deterministic ordering and the much maligned DSL

January 14th, 2010

This morning I came across a post entitled Puppet versus Chef: 10 reasons why Puppet wins.  The post attempts to explain the differences between Chef and Puppet and why Puppet is superior.  The post wasn’t great IMHO, personally I thought it was fairly poorly reasoned and made some, potentially accurate, but throughly unsubstantiated claims.

Leaving aside the issues with the post itself though, it did prompt an interesting comment thread, particularly comments  between Opcode’s CTO Adam Jacob and Reductive Lab’s Teyo Tyree (links are to the respective comments – Adam’s and Teyo’s reply).

I’m going to quote Teyo’s comment in full because I think it answers a lot of question that people have had about some of the key differences between Puppet and Chef – dependency modelling and DSL:

There is a misstatement in your assessment of Puppet’s dependency handling. You express Chef’s ordering as deterministic and imply that Puppet is in someway non-deterministic. This is not the first time you have implied this publicly, so I thought I should bring some clarity to your misstatement. The actual differentiation is procedural ordering versus a dependency graph. Puppet provides a graphing model for ordering versus a procedural model. Sure, you get procedural ordering for free with Ruby, it seems easier, and I am aware that this was a design decision for you guys. We also know that you were frustrated by “having” to express dependencies in Puppet in order to ensure consistent ordering. Properly expressed dependencies in Puppet provide ordering where you care to have it. Procedural ordering is implicit even if you don’t care. This is a BIG difference, perhaps the fundamental difference between Puppet and Chef and one that was designed into Puppet because of experiences we had trying to cope with a large code base of procedurally order scripts to manage an enterprise infrastructure. Yeah we were using make, yeah that was crazy, crazy but informative.

Your omission is related to your design decision to avoid dependency graphing, which you yourself have admitted has some major downsides, namely the inability to provide a reasonable dry-run mode, http://bit.ly/4Gcz7G. Frankly, I don’t know how you develop with out a dry-run mode, but hey I am a sysadmin not a developer.

Without a graph of resource dependencies, we would have no way of separating concerns. Consider the use case of implementing security standards. Ideally, you would want any given configuration run to bring your system into complete compliance. That sounds great but would you really want security policies not to be implemented because some earlier procedure was unable to succeed, say because it was pulling in data from a source that was not available.

So here is the difference in a nutshell. Puppet generates a catalog of dependent resources. This catalog is shipped to the clients and acted on by the Resource Abstraction Layer (RAL). On the other hand Chef, ships the required Ruby code for any node’s configuration and orders the execution of that code procedurally. These are the core differences. The DSL issues become moot if you consider Shadow Puppet or the Ruby DSL that we are developing as part of Puppet’s next release. The real difference, and IMHO Puppet’s advantage is our resource model and it’s dependency graphs versus a monolithic procedural chunk of Ruby code delivered to every client.

Here are some derived advantages of our model and a little love for the much maligned declarative external DSL:

1) Graphing base branch independence.

Parts of a catalog can be implemented more often than others. That is to say, we can tag certain resources to be checked and reconfigured more often. Additionally, parts of a configuration can be meaningfully checked but not acted on (See Adam’s discussion of noop http://bit.ly/4Gcz7G). Our customers/community love this and without a graph I don’t see how it is possible.

2) Cross host dependencies.

Our data model passes dependencies into our catalog caching system, so cross host dependencies can be resolved as well. This isn’t currently available but the framework exists and we intend to take advantage of it.

3) Failures are contained.

Critical parts of a configuration run are not excluded because of non-dependent failures.

4) Low barrier to entry for non-rubyist.

Non-rubyist can take advantage of the specification language out of the gate and Ruby developers can take advantage of the current Puppet plugin API and the future Ruby DSL, so everyone gets to use their favorite hammer.

5) Don’t like our DSL, don’t like Ruby?

Because we are only generating a catalog from the configuration language it should be fairly straight forward for anyone to generate a catalog using whatever language they choose and the RAL would be able to act on it. Come on Python people you know you want to generate catalogs with Python.

6) I can’t run Ruby on my switch!

Because we are using a data model for resources, devices that can’t/don’t have access to Ruby could still use the catalog as a basis for configuring themselves. Routers, switches, firewalls, could all be configured using the same specification language independent of how the specification is implemented, but with the resource model intact.

Finally, I think that you were a little hard on John about his comments on Chef being Rails focused. Certainly he misspoke, but the truth is that Chef development has been focused on web-application rollout in fairly homogeneous environments. Sure you can use Chef to manage the initial deployment of a web application, but in environments where you may have lots of teams utilizing compute resources for various application architectures, Puppet’s resource model shines. Security administrators can develop their Puppet manifests and not need to worry that security policies are not going to be applied because the DBA teams manifests failed. Operations teams can run Puppet in noop mode persistently and be notified if their configuration is out of compliance. Developers can make sure that the infrastructure they need for successful application deployment is available without having to worry that the security policy failed to be applied. Everyone gets to be friendlier with one and other and perhaps even get to the pub earlier on occasion.

Disclosure & Disclaimer – I am Puppet’s Release Manager and obviously heavily involved with the Reductive Lab’s team and the project.  My opinions are my own and not representative of my employer or Reductive Labs.

2 comments »

Posted in Blog, puppet

Tags:

Puppet 0.25.3 – “Clifford” released!

January 12th, 2010

CliffordPuppet 0.25.3 – code-named “Clifford”

The 0.25.3 release is a maintenance release in the 0.25.x branch.  The release addresses a regression introduced in 0.25.2 that caused issues with command execution.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.3.tar.gz

http://reductivelabs.com/downloads/puppet/puppet-0.25.3.gem

http://gemcutter.org/gems/puppet

Please note that all final releases of Puppet are signed with the Reductive Labs key.

http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying-puppet-downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.3.

CHANGELOG

* Bug #1464: Mount resource complains about missing options field
* Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #2887: Service (init) does not seem to work with require properly
* Bug #3013: util.rb:execute broken on Ruby <1.8.3
* Bug #3025: apt and aptitude providers dont work on Debian Lenny puppet 0.25.2 from gems

No comments »

Posted in Blog, open source, puppet

Tags:

Puppet 0.25.2 “Zoe” released!

January 5th, 2010

Zoe the Muppet

Puppet 0.25.2 – code-named "Zoe"

The 0.25.2 release is a significant maintenance release (123 tickets closed!) in the 0.25.x branch.

Thanks to all who contributed to the release and tested fixes – especially (but not limited to!) Peter Meier (duritong), R.I. Pienaar (Volcane), Mark Plaskin, Dan Bode, Alan Harder, Ricky Zhou, Christian Hofstaedtler, Todd Zullinger, Till Mass, Nigel Kersten, and especially Markus Roberts and Jesse Wolfe who worked around the clock to get the release out the door.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.2.tar.gz
http://reductivelabs.com/downloads/puppet/puppet-0.25.2.gem

Please note that all future final releases of Puppet will be signed with the Reductive Labs key.  Unfortunately, I am travelling and unable to access to the box with the release key on it or its backup.  A signature will be generated for this release early next week when I return to Australia.

http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying-puppet-downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.2.

RELEASE NOTES

* When setting aliases using the host type now use the host_alias attribute rather than alias.

* Puppet now has the "manage_internal_file_permissions" option which allows you to enable or disable Puppet management of internal files, for example those in /var/lib/puppet.  When "false" Puppet will NOT manage these files.  Default is "true".

* Cron type now supported on AIX

* Mailist type is now working again

* File serving permissions error messages enhanced

* SELinux now supports contexts with upper case titles

* When running the tests you no longer need to use RSpec version 1.2.2 but rather versions including and newer than.

* The debug format message has been changed and clarified from:

debug: Format s not supported for Puppet::FileServing::Metadata; has not implemented method 'from_s'

to:

debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson

* Puppetdoc now works with Regex node names

* There are now valid and proper OIDs in the LDAP puppet.schema that are unique and registered for Puppet.

* Packagers please note updated man pages including a new page for puppetqd

*    Fix for temporary file issues (https://bugzilla.redhat.com/show_bug.cgi?id=502881)

CHANGELOG

Full list of closed tickets.

 

No comments »

Posted in Blog, open source, puppet

Tags:

Photobooth James

January 3rd, 2010

No comments »

Posted in Blog

Tags: