In the tests we demonstrated you could potentially end up with some quite repetitive scenarios, for example:

  Scenario: Visiting home page
    When I go to http://www.google.com
    Then the request should succeed

If we have multiple websites we can end up with repetitive scenarios like:

  Scenario: Visiting home page
    When I go to http://www.google.com
    Then the request should succeed
  Scenario: Visiting home page
    When I go to http://www.nextsite.com
    Then the request should succeed
  Scenario: Visiting home page
    When I go to http://www.anothersite.com
    Then the request should succeed

We can summarise this series of scenarios with a nifty bit of refactoring using some Gherkin (Cucumber’s DSL) syntax called an Outline.  Using an outline we can create a kind of template like so:

Scenario Outline: Home
  When I go to <url>
  Then the request should succeed
  Examples:
  | url            |
  | http://www.google.com |
  | http://www.nextsite.com |
  | http://www.nextsite.com |

Cucumber turns each example—each table row—into a concrete scenario before looking for matching step definitions.  And hey presto when it runs you get:

$ bin/cucumber --require features/ features/home/outline.feature
Scenario Outline: Home            # features/home/outline2.feature:1
When I go to <url>              # features/steps/webrat_steps.rb:1
Then the request should succeed # features/steps/result_steps.rb:13

Examples:
| url                     |
| http://www.google.com   |
| http://www.nextsite.com |
| http://www.nextsite.com |

3 scenarios (3 passed)
6 steps (6 passed)
0m3.123s

Update – you need cucumber-nagios version 0.7.2 later to use outlines…

Leaving aside the issues with the post itself though, it did prompt an interesting comment thread, particularly comments  between Opcode’s CTO Adam Jacob and Reductive Lab’s Teyo Tyree (links are to the respective comments – Adam’s and Teyo’s reply).

I’m going to quote Teyo’s comment in full because I think it answers a lot of question that people have had about some of the key differences between Puppet and Chef – dependency modelling and DSL:

There is a misstatement in your assessment of Puppet’s dependency handling. You express Chef’s ordering as deterministic and imply that Puppet is in someway non-deterministic. This is not the first time you have implied this publicly, so I thought I should bring some clarity to your misstatement. The actual differentiation is procedural ordering versus a dependency graph. Puppet provides a graphing model for ordering versus a procedural model. Sure, you get procedural ordering for free with Ruby, it seems easier, and I am aware that this was a design decision for you guys. We also know that you were frustrated by “having” to express dependencies in Puppet in order to ensure consistent ordering. Properly expressed dependencies in Puppet provide ordering where you care to have it. Procedural ordering is implicit even if you don’t care. This is a BIG difference, perhaps the fundamental difference between Puppet and Chef and one that was designed into Puppet because of experiences we had trying to cope with a large code base of procedurally order scripts to manage an enterprise infrastructure. Yeah we were using make, yeah that was crazy, crazy but informative.

Your omission is related to your design decision to avoid dependency graphing, which you yourself have admitted has some major downsides, namely the inability to provide a reasonable dry-run mode, http://bit.ly/4Gcz7G. Frankly, I don’t know how you develop with out a dry-run mode, but hey I am a sysadmin not a developer.

Without a graph of resource dependencies, we would have no way of separating concerns. Consider the use case of implementing security standards. Ideally, you would want any given configuration run to bring your system into complete compliance. That sounds great but would you really want security policies not to be implemented because some earlier procedure was unable to succeed, say because it was pulling in data from a source that was not available.

So here is the difference in a nutshell. Puppet generates a catalog of dependent resources. This catalog is shipped to the clients and acted on by the Resource Abstraction Layer (RAL). On the other hand Chef, ships the required Ruby code for any node’s configuration and orders the execution of that code procedurally. These are the core differences. The DSL issues become moot if you consider Shadow Puppet or the Ruby DSL that we are developing as part of Puppet’s next release. The real difference, and IMHO Puppet’s advantage is our resource model and it’s dependency graphs versus a monolithic procedural chunk of Ruby code delivered to every client.

Here are some derived advantages of our model and a little love for the much maligned declarative external DSL:

1) Graphing base branch independence.

Parts of a catalog can be implemented more often than others. That is to say, we can tag certain resources to be checked and reconfigured more often. Additionally, parts of a configuration can be meaningfully checked but not acted on (See Adam’s discussion of noop http://bit.ly/4Gcz7G). Our customers/community love this and without a graph I don’t see how it is possible.

2) Cross host dependencies.

Our data model passes dependencies into our catalog caching system, so cross host dependencies can be resolved as well. This isn’t currently available but the framework exists and we intend to take advantage of it.

3) Failures are contained.

Critical parts of a configuration run are not excluded because of non-dependent failures.

4) Low barrier to entry for non-rubyist.

Non-rubyist can take advantage of the specification language out of the gate and Ruby developers can take advantage of the current Puppet plugin API and the future Ruby DSL, so everyone gets to use their favorite hammer.

5) Don’t like our DSL, don’t like Ruby?

Because we are only generating a catalog from the configuration language it should be fairly straight forward for anyone to generate a catalog using whatever language they choose and the RAL would be able to act on it. Come on Python people you know you want to generate catalogs with Python.

6) I can’t run Ruby on my switch!

Because we are using a data model for resources, devices that can’t/don’t have access to Ruby could still use the catalog as a basis for configuring themselves. Routers, switches, firewalls, could all be configured using the same specification language independent of how the specification is implemented, but with the resource model intact.

Finally, I think that you were a little hard on John about his comments on Chef being Rails focused. Certainly he misspoke, but the truth is that Chef development has been focused on web-application rollout in fairly homogeneous environments. Sure you can use Chef to manage the initial deployment of a web application, but in environments where you may have lots of teams utilizing compute resources for various application architectures, Puppet’s resource model shines. Security administrators can develop their Puppet manifests and not need to worry that security policies are not going to be applied because the DBA teams manifests failed. Operations teams can run Puppet in noop mode persistently and be notified if their configuration is out of compliance. Developers can make sure that the infrastructure they need for successful application deployment is available without having to worry that the security policy failed to be applied. Everyone gets to be friendlier with one and other and perhaps even get to the pub earlier on occasion.

Disclosure & Disclaimer – I am Puppet’s Release Manager and obviously heavily involved with the Reductive Lab’s team and the project.  My opinions are my own and not representative of my employer or Reductive Labs.

The typical enterprise monitoring and configuration management set-up is generally something like this: central server(s) manage and monitor a number of services on local and/or remote hosts.  Digging down, for a web server these checks might be something like:

• Is the Apache package installed and the appropriate version?
• Is the Apache service running?
• Can I connect to the HTTP port and is HTML returned?

Multiply this by a few hundred iterations of hosts and types of services and you're probably looking at your typical Nagios, Puppet, Cfengine, Hyperic, Tivoli or Patrol set-up. Add logging, alerts, graphs and reporting and this is probably pretty close to the environment that most system administrators manage and monitor every day. All the bases covered, appropriate alerts when things go down, reporting for your management, etc, etc.

So that's all good right and we don't need to do anything more?  Nope, not quite.  All this monitoring misses something critical – we're not actually monitoring that the service does what it should.  Yes, it matters whether Apache is installed, the Apache service is running, and you can connect to HTTP but does this actually prove anything about the availability of the service we're managing and providing for our customers?  No again.  You can connect to the port, have the service running and still not be delivering the right content or providing the appropriate functionality to the customer. And ultimately that's what our jobs are all about – delivering service to the customer.  Whether internal ("the business") or an external customer, they don't care about the infrastructure.  Nor the technology, its configuration or anything else about the widgets that deliver the services they use.  They just want "technology" to be:

1. Available,
2. Functional, and
3. Cost-effective.

To deliver (and measure) the first two items on that list we're going to need more than just a check that says the Apache server is up.  We need to demonstrate that the service delivered by that infrastructure was available to our customers AND functioning as intended.  If it isn't functioning as intended, all the availability in the world is meaningless because the customer isn't getting what they want.

(Needless to say most enterprise monitoring measures of "availability" are bogus.  Using an ICMP ping of a host, uptime or checking a process as a measure of availability merely demonstrates that the asset is up.  It doesn't demonstrate that the asset is performing the function it should hence doesn't actually measure "availability".)

All is not lost though, we have the technology, we can rebuild your monitoring environment: better, stronger and more relevant.  How?  By stealing someone else's idea.  You see developers face the same challenge of delivering appropriate functionality.  In their case an application may compile and run but produce incorrect output or worse no output at all.  Like our monitoring, this leaves our developer short on knowing whether they are delivering functionality to the customer. So to ensure that their applications do what they have promised, developers test them. 

There are lots of different kinds of testing: functional tests to confirm things work, performance testing, user acceptance tests to ensure user experience is suitable.  But one kind of testing has become increasing important: behavioural testing.  Behavioural testing checks that each function, method or procedure not only works but behaves in the intended way. Developers call this methodology Behaviour Driven Development or Test Driven Development (BDD and TDD for short). In a BDD/TDD environment each component of your code is tested to ensure it is behaving correctly.  The basic element of this testing is called a unit test.  In BDD, unit tests are developed for each function to determine whether it is fit for use.  Let's look a simple example of a function, one that adds numbers, and a unit test to confirm its behaviour is correct.  We'll start by articulating our function (in pseudo code).

def addition(val1,val2)
  print "Total =" val1 + val2
end

So if our function compiles, executes and returns a result does that mean it works?  No, because we can't guarantee it returns the right result.  To overcome this gap in our knowledge, we devise a simple unit test (again in pseudo code).

test addition
  val1 = 4
  val2 = 6
  total = 10

  result = addition(val1,val2)
  if result != total then print "Function addition failed - incorrect total"
end

In our test we first set the input values and what the resulting output should be (in the object-orientated world these are called "mock" objects and are designed to simulate real objects in a controlled way).  We then run the function we'd like to test and check that the returned result matches the mock output.  If the result doesn't match then we return an error message and we know we need to fix the function. This combination of functional tests and behavioural testing means that not only do we ensure our applications runs but that when it does run it is behaving correctly.

So can we apply Behaviour Driven Development to our infrastructure to test that it is behaving correctly?  Enter Behaviour Driven Infrastructure. Behavioural Driven Infrastructure or BDI applies the principles of Behavioural Driven Development to the management of infrastructure.  And boy is it cool. Let's jump right in and see what a BDI check might look like.  Remember our Apache checks?  Let's design some simple behavioural checks to supplement these, checks that step up from monitoring our infrastructure into monitoring the behaviour of our service:

• The site contains some static value or content
• The site contains some dynamic content that can be validated, for example data drawn from a database
• When I click or follow links I get the right pages returned
• When I fill in a form the values are validated
• When I press a button the form is submitted
• When I select a field or drop-down the right values are populated

Notice the key difference between the checks we defined earlier and these checks?  These checks involve the behaviour of the service rather than its binary status.  Instead of the site being "up" we're testing that the site returns the right content or in other words that the site behaves correctly. We can see that determining tests for a website is relatively easy but what about other types of infrastructure and services?  You can develop similar tests for a wide variety of infrastructure:

• SSH – check that a particular user can login, an inappropriate user fails and is logged or alerted
• SMTP – check that the daemon receives an email and delivers it, check it rejects mail it should reject, check authentication works
• IMAP – Check you can receive email from a mailbox, check authentication works
• MySQL/database/LDAP/directories – check you can query a record and that the record returned is correct
• Load balancer – check connections are switched between hosts
• DNS – check output of DNS queries is correct
• Backups – backup and restore a file
• Nagios/Enterprise Monitoring – Check tests pass, fail, escalate, send notifications
• Samba/NFS – create, change, delete a file
• Sudo – check you can run a sudo command and check inappropriate sudo commands fail and log

Notice what we're trying to do?  We're testing that the service does what the customer expects it to do.  This not only proves the service is behaving the way we want it to but also demonstrates that the service is available.

So all of these tests sound easy when written down like this but how do we implement them? We're going to articulate our BDI tests in plain English using a tool called Cucumber and also introduce you to a spin-off tool called Cucumber-Nagios (which I talked about in a previous post). 

Cucumber is a behavioural testing framework used heavily in the Ruby community (and in the Java, .NET, Flex communities too) that is simple and easy to learn – even for non-developers. Cucumber-Nagios takes this a little further to combine Cucumber with built-in testing frameworks (web using Webrat and SSH using Net::SSH for example) and outputs test results as Nagios check data.  Perfect for immediate integration into your existing enterprise monitoring solution (and easily hack'able to output as other data formats also).  The beauty of Cucumber-Nagios is that it comes with pre-built tests components that we can adapt to suit our environment. Cucumber has two components:

1. Plain text tests called "features" which contain the different scenarios we want to test, and
2. Supporting code called "steps" to actually test each "feature" and its associated scenarios.

Let's create a simple behavioural test for our website.  We start by installing Cucumber-Nagios via a Gem:

$ sudo gem install cucumber-nagios

On some distributions you may need to install gemcutter first:

$ sudo gem install gemcutter

$ sudo gem tumble

The cucumber-nagios gem will install the cucumber-nagios-gen binary which we will use to create a project to hold our tests.

$ cucumber-nagios-gen project test_project

Here we've told the cucumber-nagios-gen binary to create a project called test_project. A project is a mini-application that contains the right directory structure and files to run our tests. Change into the resulting directory:

$ cd test_project

We then need to bundle some supporting gems into the project to allow it to be self-contained:

$ gem bundle

Now we have a local version of cucumber-nagios-gen installed in the bin directory of the project and we can use this to create some features to test:

$ bin/cucumber-nagios-gen feature www.google.com content

This creates a feature in a file called content.feature (each Cucumber file can contain one feature and must have a suffix of .feature).  Let's open this file and examine our feature:

Feature: www.google.com
  It should be up

  Scenario: Visiting home page
    When I go to http://www.google.com
    Then the request should succeed

Cucumber uses a business-readable domain-specific language called Gherkin to write its features. Let's deconstruct what each section of our feature means:

 Feature: Some terse yet descriptive text of what is desired
   In order to realize a named business value
   As an explicit system actor
   I want to gain some beneficial outcome which furthers the goal

   Scenario: Some determinable business situation
     Given some precondition
       And some other precondition
     When some action by the actor
       And some other action
       And yet another action
     Then some testable outcome is achieved
       And something else we can check happens too

   Scenario: A different situation
       ...

The feature starts with a description, in our case www.google.com and then some text that describes the business value of the feature, that the website should be up.  Each scenario used to validate that business value is then listed, each with it's own description and a series of steps that they involve.  There are three types of steps – Given, When or Then:

1. Givens – put the system into a known state
2. Whens – describe the key action that is being performed
3. Thens – observe the outcomes

Which can be summarised as: Given some condition When I do this action Then I will see this outcome.

Each step in our scenario has to start with one of these types (but you don't need to use all of them) and you can see in our example feature we're using a When and a Then:

 When I go to http://www.google.com
 Then the request should succeed

Simple and plain English.  There is a bit more to Gherkin that we haven't touched on (but you can read about at that link) but let's try to run our feature now using the cucumber-nagios binary (obviously you need to be connected to the Internet for the feature to work appropriately):

$ bin/cucumber-nagios features/www.google.com/content.feature
Critical: 0, Warning: 0, 2 okay | passed=2, failed=0, nosteps=0, total=2

We can see that we've selected and executed our feature and it has returned some Nagios plug-in output (which appears as Critical, Warning, or Okay) and that 2 steps are Okay (or passed). 

But wait a second, how have they passed?  We haven't written any code at all and it works?  Well as I mentioned Cucumber-Nagios contains a set of pre-defined steps for a variety of common tasks.  You can use these steps and not have to write any code.  Let's look at the associated step we've just used.  This pre-defined step is contained in the features/steps/webrat.steps file in our project:

When /^I go to (.*)$/ do |path|
  visit path
end

You can see it's a very simple bit of code that uses a regular expression to check our feature file for some specific language, in this case the words "I go to URL".  The regular expression captures a URL and passes it to Webrat which runs the visit function and returns the result.  This is passed to the next step:

Then the request should succeed

Cucumber-Nagios also contains a set of pre-defined steps for handling the results, the Then steps.  These steps are contained in the features/steps/result_steps.rb file.  In this case we've used the following step:

Then /^the (.*) ?request should succeed/ do |_|
   success_code?.should be_true
end

This step checks the result of the When step and if it registered a success then the step passes and returns an Okay result.

Now, let's see how we can add another scenario to our feature.

Feature: www.google.com
  It should be up
  You should be able to click on the Videos link

  Scenario: Visiting home page
    When I go to http://www.google.com
    Then the request should succeed

  Scenario: Clicking on the Videos link
    When I go to http://www.google.com
      And I follow "Videos"
    Then I should see "Google Videos"

In our new scenario we've tested following a link on the Google site to the Google Videos site.  We've also used another piece of Cucumber statement, And, which is a cleaner way of writing multiple Given-When-Then steps.

This new When step also uses a pre-defined step from Cucumber-Nagios:

When /^I follow "(.*)"$/ do |link|
   click_link(link)
end

And a pre-defined Then step:

Then /^I should see an? (\w+) message$/ do |message_type|
  response.should have_xpath("//*[@class='#{message_type}']")
end

We can then run our new scenario:

$ bin/cucumber-nagios features/www.google.com/content.feature
Critical: 0, Warning: 0, 5 okay | passed=5, failed=0, nosteps=0, total=5

And see that we've now got 5 steps that pass, including our 3 new steps.  Simple eh?  And readily extensible.

So we've seen that Cucumber and BDI allows infrastructure testing but we can also extend this concept further.  So we've seen that Cucumber can establish that the service is behaving correctly but what if we also used it to test for business logic and business rules also?  Let's take a simple example, expressed here as a feature:

Feature: homeloansite.com
  It should be up
  And I should be able to find the current Rate

  Scenario: Checking the current interest rate
    When I visit "http://homeloansite.com"
    Then I should see the Rates section
    Then I should see current Rate
      And the current Rate should equal 5%

You'd need to write an additional step to return the Rate value in Cucumber-Nagios to get this example to work but assuming we had we could then run our new feature:

$ cucumber-nagios features/homeloansite.com/homeloan.feature
Critical: 0, Warning: 0, 4 okay | value=4.000000;;;;

In this feature we've checking four steps:

1. That the Home Loan site is up
2. That it has a section called "Rates"
3. That the section contains a Rate, and
4. That the Rate equals 5%

So in this single scenario we've killed a lot of birds with one cucumber.  We've confirmed the site is up, we've confirmed that the right content is being delivered and lastly we've confirmed that the information being delivered is correct.  This goes beyond our traditional infrastructure testing to confirm that a piece of business data, our home loan rate, is correct.  Using this test we could then configure Nagios to alert the appropriate business contact that the website was displaying the wrong rate.  And hey presto we're delivering business value.

This feature is just the tip of the iceberg.  We could do a whole collection of other things, for example test transaction limits for a finance application, confirm access controls for applications, check the output of reports, and anything else where business logic or rules are exposed and can be exercised and tested.

Lastly, it's important to remember we're not replacing our traditional checks when using Cucumber.  We still need to watch the low level components we're just adding another, more powerful and insightful layer to our monitoring.  A layer that more accurately represents the value that you offer to the business than graphs of ICMP ping results.

Configuration management tools are finally coming into their own. Powerful, automated infrastructure management is now available in a wide variety of open source tools. Tools written in different languages, using varying operational methodologies and embracing differing philosophies. Come meet some of the creators and maintainers of these cutting edge tools like cfengine, Puppet, AutomateIT, Chef, and bcfg2 and quiz them in the why and hows of their tools and the philosophies behind them.

The panel includes the following cool participants:

- Luke Kanies from Reductive Labs for Puppet
- Brendan Strejcek of Cfengine
- Adam Jacob from Opscode for Chef
- Narayan Desai of bcfg2
- Igal Koshevoy of AutomateIt

If you have any interest in CM or operations automation I strongly recommend you attend.