Archive for February, 2010

What DevOps means to me…

February 19th, 2010

Over the last year or so a bunch of presumptuous European sysadmins and developers, joined by some of their American brethren and even a couple of us antipodeans (there are others too!) have been talking about a concept called .  is the merger of the realms of development and operations (and if truth be told elements of product management,QA, and *winces* even sales should be thrown into the mix too).

The Broken

So … why should we merge or bring together the two realms?  Well there are lots of reasons but first and foremost because what we’re doing now is broken.  Really, really broken.  In many shops the relationship between development (or engineering) and operations is dysfunctional to the point of occasional toxicity.

Here’s an example I think everyone will be at least partially familiar with: the minefield that is project to production software deployment.  Curse along as I explain.

Development builds an application, the new hotness which promises customers all the whizz-bang features and will make the company millions.  It is built using cutting edge technology and a brand new platform and it has got to be delivered right now.  Development cuts code like crazy and gets the product ready for market ahead of schedule.  They throw their masterpiece over the fence to Operations to implement and dash off to the pub for the wrap party.

Operations catches the deployment and is filled with horror.

The Operations team summarises their horror and says one or more of:

  • The wonder application won’t run on our infrastructure because {it’s too old, it doesn’t have capacity, we don’t that version}
  • The architecture of the application doesn’t match our { storage, network, deployment, security } model
  • We weren’t consulted about the { reporting, security, monitoring, backup, provisioning } and it can’t be “productionised”.

But Operations persevere and install the new hotness – cursing and bitching throughout.  Sadly, after forcing the application onto infrastructure and bending and twisting the architecture to get it running, the performance of the new application can be summed up as “epic fail”.

Operations sighs and starts logging problems and passing issues back to the Development team.  Their responses generally come from the following pool:

  • It’s not our fault – our code is perfect – it’s just been poorly implemented
  • Operations are stupid and don’t understand the new hotness – why can’t they implement the cutting edge technology? Why are they so backward?
  • It runs fine on my machine…

The interactions between teams quickly becomes a toxic blame storm. The customers (and by extension the shareholders, investors and management) then become the losers.  The loop gets closed with the company losing bucket loads of money and everyone losing their jobs.  EPIC and FAIL.

What’s different about ?

is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups. In a environment, developers and sysadmins build relationships, processes, and tools that allow them to better interact and ultimately better service the customer.

is also more than just software deployment – it’s a whole new way of thinking about cooperation and coordination between the people who make the software and the people who run it.  Areas like automation, monitoring, capacity planning & performance, backup & recovery, security, networking and provisioning can all benefit from using a model to enhance the nature and quality of interactions between development and operations teams.

Everyone in the community has a slightly different take on “What is ?”  We all bring different experiences and focuses to the problem space.  I personally see as having four quadrants:

Simplicity

KISS is King and in that vein this section is simple too. Design simple, repeatable, and reusable solutions. Simplicity saves documentation, training, and time.  Simplicity increases the speed of communication, avoids confusion, and helps reduces the risk of development and operational errors.  Simplicity gets you to the pub faster.

Relationships

Engage early, engage often. Development teams need to embed operations people into their project and development life cycles.  Invite operational people to your scrum or development meetings.  Share ideas and information about product plans and new technologies. Gather operational requirements when gathering functional ones. As a project progresses test deployment, backup, monitoring, security and configuration management as well as application functionality.  The more issues you fix during the project the less issues you expose your customers to when the application is live.  Educate operations people about the applications architecture and the code base. The more information operations people can feed you about a problem with the code the less trouble-shooting you need to perform and the faster the problem can be fixed.

Operations people need to bring development people into the problem and change management space. Invite developers into your team meetings. Share your roadmaps and upgrade plans.  Understand where future development is heading to better ensure infrastructure deployments match product requirements.  Developers also bring skills, knowledge and tools that can make your environment easier to manage, more efficient and cleaner. Learn to code or if you’re a hack-n-slash systems programmer like me then learn to code better. Concepts like building tools with APIs rather than closed interfaces, distributed version control, test driven development, and methodologies like Agile Development, Kanban and Scrum can revolutionise operational practises in the same way they’ve changed the way code is cut.

Don’t be afraid of ideas and approaches from outside your domain – we can all learn things, even if it’s “let’s never do it that way again…!”, from how others do things and ultimately? Guess what? Yep, we’re all on the SAME team.

Remember that interactions between people rank, in decreasing order of effectiveness (in IMHO but backed by some research):

  1. Face to face
  2. Video conference
  3. Phone
  4. IM & IRC
  5. Email

Process

Don’t underestimate the power of process and automation.  Many shops do process engineering – ranging from hand-written lists to ISO9001. Those processes generally have one key flaw: they focus on the outcome and its inevitability.  A simple process might provision a host – Step 1 install machine, Step 2 cable machine, Step 3 install OS, etc, etc. Assuming all goes to process then at the end of Step x you will have a fully provisioned host. But what happens if it doesn’t go right?  If your process breaks or you receive some anomalous output how does your process deal with it?  Instead think about process as a journey and map out the potential pitfalls and obstacles.  Treat your processes like applications and build error handling into them.  You can’t predict every application or operational pitfall or issue but you can ensure that if you hit one your process isn’t derailed.

Link process together across domains – software deployment, monitoring, capacity planning and other “operational” processes have their start in the development world.  Software deployment is the logical conclusion of the software development life cycle and should be viewed as such rather than a separate operational process. Another example is metrics and monitoring, it is hard to measure anything  without understanding the baselines and assumptions made in the development domain.  Joint processes also mean more opportunity for development and operations interaction, understanding and joint accountability. Finally, joint process development means single repositories for documentation and other opportunities for economies of scale.

Automate, automate, automate. Build or make use of simple and extensible tools (make sure they have APIs and machine readable input and output – see James White’s Infrastructure Manifesto).  Use tools like Puppet (or others) to manage your configuration.  Remember to extend your automation umbrella cross-domain and end-to-end in your environment – manage development, testing, staging and production environments with the same tools and processes.  Not only does this have economies of scale benefits in and management but it means you can test deployment and management alongside functionality as your application and new codes rolls toward production.

Finally, when building process and automation always keep the KISS principle in mind. Complexity breeds opportunities for error. Build simple processes and tools that are easy to implement, manage and maintain.

Continuous Improvement

Don’t stop innovating and learning.  Technology moves fast.  So do customer requirements. Build continuous improvement and integration into your tools and processes.  Here is a good place operations people can learn from (good) developers about practises like test-driven development.  A good example here is to build tests for your software deployment process and infrastructure.  They are often an application in their own right and should be developed and maintained correctly. Your monitoring could also be extended with behavioural testing to deliver better business value.  Look at using development domain tools, like Hudson for example, to explore and measure the operational domain.

Learn from mistakes and from outages.  Seek root cause aggressively AND cross-domain.  If you have an outage and a post-incident review then bring development and operational teams together to review the incident.  Sometimes some simple code refactoring can save making infrastructure changes.  Work together to fix root cause, treat it with the same process you develop to conduct project to production software deployment, rather than relegating them to incident review reports or batting issues between teams.

Me

Finally, for me is about people and nature of the environment you want to work in.  The best thing about the movement for me is that it is trying to foster behaviours and environments where people work together towards joint goals rather than at cross-purposes or at odds.  That’s a world I’d much rather use my skills in.

7 comments »

Posted in Blog, devops, puppet

Tags:

Puppet ParsedFile types and providers

February 13th, 2010

In a recent post I talked about how easy it is to generate types and providers. In that post I used the example of a very simple Subversion and repository type, called repo. I’d like to show another example of a type and , this one used to manage the contents of the /etc/shells file. This type and makes use of some built-in functionality that allows the simple parsing of files and the management of their contents. To do this has a called that can be included into your own providers to provide this functionality.

Let’s start with our type:

Puppet::Type.newtype(:shells) do
    @doc = "Manage the contents of /etc/shells
 
            shells { "/bin/newshell":
                ensure => present,
            }"
 
    ensurable
 
    newparam(:shell) do
        desc "The shell to manage"
 
        isnamevar
 
    end
 
    newproperty(:target) do
        desc "Location of the shells file"
 
        defaultto {
            if
                @resource.class.defaultprovider.ancestors.include? (Puppet::Provider::ParsedFile)
                @resource.class.defaultprovider.default_target
            else
                nil
            end
        }
    end
end

So – pretty simple. We create a block ::Type.newtype(:shells) do that creates a new type, which we’ve called shells. Inside the block we’ve got a @doc string. This is the documentation for the type. Add whatever level of detail and examples in here that is required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. types use the ensure property to determine the state of a configuration item. In our previous example, ensurable resulted in three methods in the : create, destroy, and exists?. In a we don’t use these methods at all as we’ll see shortly but rather specify how to handle each record in the file.

We’ve defined a new parameter – this one called shell.

newparam(:shell) do
        desc "The shell to manage"
 
        isnamevar
end

The shell parameter is the shell we’re going to manage in the /etc/shells file. We’ve also used another piece of automagic, isnamevar, to make this parameter the “name” variable for this type. In -speak, the value of this parameter is used as the name of the resource.

Lastly in our type we’ve specified an optional parameter, target, that allows us to override the default location of the shells file, usually /etc/shells.

newproperty(:target) do
        desc "Location of the shells file"
 
        defaultto {
            if
                @resource.class.defaultprovider.ancestors.include? (Puppet::Provider::ParsedFile)
                @resource.class.defaultprovider.default_target
            else
                nil
            end
        }
end

The target parameter is optional and would only be specified if the shells file wasn’t located in the /etc/ directory. It uses the defaultto structure to specify that the default value for the parameter is the value of default_target variable in the .

The for our type is also very simple:

require 'puppet/provider/parsedfile'
shells = "/etc/shells"
 
Puppet::Type.type(:shells).provide(:parsed, :parent => Puppet::Provider::ParsedFile, :default_target => shells, :filetype => :flat) do
 
    desc "The shells provider that uses the ParsedFile class"
 
    text_line :comment, :match => /^#/;
    text_line :blank, :match => /^\s*$/;
 
    record_line :parsed, :fields => %w{name}
end

The shells is stored in a file called parsed.rb in a directory named for the in the directory, for example:

/usr/lib/ruby/site_ruby/1.8//type/shells.rb
/usr/lib/ruby/site_ruby/1.8///shells/parsed.rb

The file needs to be named parsed.rb to allow to load the .

We first include the code at the top of our , require '//' and set a variable called shells to the location of the /etc/shells file. We’re going to use this variable a bit later.

Then we tell that this is a called shells. We specify a :parent value that tells that this should inherit the and make its functions available. We then specify the :default_target value to the shells variable we’ve just created. This tells the , that unless it is overridden by the target attribute, that the file to act upon is /etc/shells.

Then we use a desc method that allows us to add some documentation to our .

The next lines are the core of the . They tell the how to manipulate the target file to add or remove the required shell. The first two lines, both text_lines, tell how to match comments and blank lines respectively.

    text_line :comment, :match => /^#/;
    text_line :blank, :match => /^\s*$/;

We specify these to let know to ignore these lines as unimportant.

The next line performs the actual parsing of the relevant line in the /etc/shells file:

    record_line :parsed, :fields => %w{name}

The record_line parses each line and divides it into fields, in our case we only have one field: name. The name in this case is the shell we want to manage. So if we specify:

shells { "/bin/newshell":
    ensure => present,

Then would use the to add the /bin/newshell by parsing each line of the /etc/shells file and checking if the newshell is present. If it is, then will do nothing. If not, then will add newshell to the file. If we changed the ensure attribute to absent then would go through the file and remove the newshell if it is present.

It is important to remember that providers do have some limitations, they aren’t good at managing complex files such as configuration files with multi-line options, they are best for simple files that contain single line lists of entries such as the cron file entries or the /etc/hosts and /etc/shells files.

You can see the complete code for this type and its providers at my Puppet repository on GitHub. Quite a lot of the existing types and providers use providers (the cron type for example) and you can use these as examples of how to create your own providers. You can also find further documentation (in a lot more detail!) on creating your own types and providers at the Puppet wiki.

No comments »

Posted in Blog, open source, puppet

Tags:

Help Vampires: A Spotter’s Guide

February 9th, 2010

This is an excellent post on the phenomena of “ vampires” who suck up the goodwill in communities.  It tells you how to spot them, how to handle them and how to protect your community against them.

No comments »

Posted in Blog, open source

Tags:

Creating Puppet types and providers is easy…

February 1st, 2010

types are used to manage individual configuration items.  has a package type, a service type, a user type, etc.  Each type has providers. Each handles the management of that configuration on a different platform or tool, for example the package type has aptitude, yum, RPM, and DMG providers (amongst 22 others – what is wrong with people that they need to invent new packaging systems… but I digress).

There are a lot of types, in fact I think covers a pretty good spectrum of configuration items that need to be managed.  I don’t know of anything in particular that is missing that I can’t live without.  But there are little gaps that are annoying, I’d like network and firewall types for example, but creating both these types in a generic enough way to multiple platforms would be, IMHO, a non-trivial problem. 

Another gap is VCS/DVCS management. A lot of people use source code in repositories to do things with (including install stuff from you bad people – package things … it’s healthier). currently relies on creating and removing these repositories with the exec type (which executes scripts or binaries), for example:

exec { "svn co http://core.svn.wordpress.org/trunk/ /var/www/wp":
    creates => "/var/www/wp",
}

This is a bit ugly and it’d be a lot easier to write a type to manage repositories. But types and providers are written in Ruby and really, really complex and hard to develop. Right? Right?

No. No, they are not… and I’m going to create a simple type and to show you. :)

Here’s a very (very!) simple type, called repo, for managing repositories. I’ve created providers for SVN and as examples also. The first part of the repo type is the type itself – these are usually stored in lib//type or distributed via modules (see the PluginsInModules page in the wiki). I’ll create a file called repo.rb.

$ touch repo.rb

And then populate the file:

Puppet::Type.newtype(:repo) do
    @doc = "Manage repos"
 
    ensurable
 
    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
 
    end
 
    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end
end

So – pretty simple. We create a block ::Type.newtype(:repo) do that creates a new type, which we’ve called repo.

Inside the block we’ve got a @doc string. This is the documentation for the type. Add whatever level of detail and examples in here that is required.

We’ve also got the ensurable statement. Ensurable provides some “automagic” that creates a basic ensure property. types use the ensure property to determine the state of a configuration item.

service { "sshd":
    ensure => present,
}

The ensurable statement tells to expect three methods: create, destroy and exists? in our . These methods, allow, respectively:

  • A command to create the resource
  • A command to delete the resource, and
  • A command to check for the existence of the resource

All we then need to do is specify these methods and their contents and creates the supporting infrastructure around them but more on this when we look at our providers.

Next, we’ve defined a new parameter – this one called source.

    newparam(:source) do
        desc "The repo source"
 
        validate do |value|
            if value =~ /^git/
                resource[:provider] = :git
            else
                resource[:provider] = :svn
            end
        end
 
        isnamevar
    end

The source parameter will tell the repo type where to go to retrieve/clone/checkout our source repository.

In this parameter we’re also using a hook called validate. Normally used to check the value for appropriateness here we’re using it to take a guess at what to use. Our code says, if the source parameter starts with then use the , if not default to the Subversion . This is obviously fairly crude as a default and we can override this by defining the attribute in our resources:

provider => git,

We’ve also used another piece of automagic, isnamevar, to make this parameter the “name” variable for this type. In -speak, the value of this parameter is used as the name of the resource.

(Types have two kinds of values – properties and parameters. Properties “do things”. They tell us HOW the works. We’ve only defined one property, ensure, by using the ensurable statement. Parameters are more like variables, they contain information relevant to configuring the resource the type manages rather than “doing things”.)

Finally, we’ve defined another parameter, path.

    newparam(:path) do
        desc "Destination path"
 
        validate do |value|
            unless value =~ /^\/[a-z0-9]+/
                raise ArgumentError , "%s is not a valid file path" % value
            end
        end
    end

This is a variable value that specifies where the repo type should put the cloned/checked-out repository. In this parameter we’ve again used the validate hook to create a block that checks the value for appropriateness. Here we’re just checking, very crudely, to make sure it looks like the destination path is a valid fully-qualified file path. We could also use this validation for the source parameter to confirm a valid source URL/location was being provided.

(You can also use another hook called munge to adjust the value of the parameter rather than validating it before passing it to the .)

And that is it for the type.

Next, we need to create a for our type. Let’s start with a Subversion like so:

require 'fileutils'
 
Puppet::Type.type(:repo).provide(:svn) do
    desc "SVN Support"
 
    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"
 
    def create
        svncmd "checkout", resource[:name], resource[:path]
    end
 
    def destroy
        FileUtils.rm_rf resource[:path]
    end
 
    def exists?
        File.directory? resource[:path]
    end
end

Up front we’ve required the fileutils library, which we’re going to use a method from. Next, we’ve defined the as a block:

Puppet::Type.type(:repo).provide(:svn) do

We tell that this is a called svn for the type called repo.

Then we use a desc method that allows us to add some documentation to our .

Next, we define the commands that this will use, here the svn and svnadmin binaries, to manipulate our resource’s configuration.

    commands :svncmd => "svn"
    commands :svnadmin => "svnadmin"

uses these commands to determine if the is appropriate to use on a client, if can’t find these commands in the local path then it will disable the .

Next, we’ve defined three methods – create, destroy and exists?. Sounds familiar? Yep, these are the methods that the ensurable statement expects to find in the :

The create method ensures our resource is created. It uses the svn command to create a repository with a source of resource[:name] (remember the source parameter in our type is also the name variable of the type – we could also specify resource[:source] here too) and a destination of resource[:path] (the value of the path attribute).

The delete method ensures the deletion of the resource. In this case, it deletes the directory and files specified in the resource[:path] parameter.

Lastly, the exists? method checks to see if the resource exists. Its operation is pretty simple and closely linked with the value of the ensure attribute in the resource:

  • If exists? is false and ensure is present, then create method will be called.
  • If exists? is true and ensure is set to absent, then the destroy method will be called.

In this case the exists? method checks if there is already a directory at the location specified in the resource[:path] parameter.

So, let’s put all this together and create a resource with our new type. I’ve assumed you’ve already distributed your type and providers to . We can then create a resource like:

repo { "wp":
    source => "http://core.svn.wordpress.org/trunk/",
    path => "/var/www/wp",
    ensure => present,
}

Simple eh? We specify a repo resource, the source we wish to check out or clone from, the destination path and the ensure attribute (present or absent) and that’s it.

You can see the complete code for this type and its providers at my Puppet repository on GitHub. It’s obviously very basic but should be easy to extend to provide additional capabilities (and currently has no tests – my bad). You can find further documentation (in a lot more detail!) on creating your own types and providers at the Puppet wiki.

14 comments »

Posted in Blog, open source, puppet

Tags: