Archive for January, 2010

Puppet 0.25.4 released!

January 29th, 2010

You wanted “release early, release often” and the Puppet team has delivered!
The 0.25.4 release is a maintenance release (with one important feature – pre/post transaction hooks – discussed below) in the 0.25.x branch.  The release primarily addresses a regression introduced in 0.25.3 that caused issues with creating cron jobs.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.4.tar.gz

http://reductivelabs.com/downloads/gems/puppet-0.25.4.gem

http://gemcutter.org/gems/puppet

Please note that all final releases of Puppet are signed with the Reductive Labs key – http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying…

Please report feedback via the Reductive Labs Redmine site: http://projects.reductivelabs.com

Please select an affected version of 0.25.4

RELEASE NOTES

Pre/Post Transaction hooks

There is a new feature in this release: pre and post transaction hooks.  These hooks allow you to specify commands that should be run pre and post a Puppet configuration transaction.   They are set with the prerun_command and postrun_command settings in the puppet.conf configuration file.

prerun_command = /bin/runbeforetransaction
postrun_command = /bin/runaftertransaction

The command must exit with 0, i.e. succeed, otherwise the transaction will fail – if the pre command fails before the transaction is run and if the post command fails at the end of the transaction.

CHANGELOG
*  Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #3001: Can’t manage broken links
* Bug #3039: 0.25.3 gem spec specifies the executables incorrectly
* Bug #3075: sshkey host aliases broken by fix for #2813
* Bug #3088: Puppetd fails to stop after receiving SIGTERM
* Bug #3089: puppetlast gsub! error
* Bug #3093: Blastwave provider broken in 0.25.3
* Bug #3104: Test failed: Puppet::Network::XMLRPCClient when performing the rpc call and an exception is  raised.should log and raise XMLRPCClientError if Timeout::Error is raised
* Bug #3112: Problem with adding and removing crons
* Bug #3122: Uncharacterized failure in fileserving under OS X
* Bug #3125: Dpkg tests failing
* Feature #2914: Transactions should have before and after hooks

The Tortoise and not the Hare 2 – Principles

January 24th, 2010

Kanban

In my first post I introduced you to the Toyota Production System and the Kanban signalling system. At the core of the TPS is the concept of maintaining efficiency and eliminating waste.  To govern this processes the TPS has a series of basic principles that articulate how this is achieved:

  1. Create continuous process flow to bring problems to the surface
  2. Use the “pull” system to avoid overproduction
  3. Level out or smooth the workload AKA “Heijunka” – be the tortoise not the hare
  4. Build a culture of stopping to fix problems, to get quality right from the first
  5. Standardized tasks are the foundation for continuous improvement and employee empowerment
  6. Use visual control so no problems are hidden
  7. Use only reliable, thoroughly tested technology that serves your people and processes.

I’ve skipped all the TPS rules around long-term thinking (for example the first principle of the TPS – Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals), corporate harmony, people development and organisational learning.  I haven’t skipped them because they aren’t important but because they aren’t immediately relevant to this discussion.  If you haven’t got the others right too though I suspect your organisation will go pear-shaped in other ways.

In each of the subsequent posts I am going to look at one of the seven principles I’ve articulated above, starting with exploring how you can make continuous process flow work for you.

Puppet, Chef, deterministic ordering and the much maligned DSL

January 14th, 2010

This morning I came across a post entitled Puppet versus Chef: 10 reasons why Puppet wins.  The post attempts to explain the differences between Chef and Puppet and why Puppet is superior.  The post wasn’t great IMHO, personally I thought it was fairly poorly reasoned and made some, potentially accurate, but throughly unsubstantiated claims.

Leaving aside the issues with the post itself though, it did prompt an interesting comment thread, particularly comments  between Opcode’s CTO Adam Jacob and Reductive Lab’s Teyo Tyree (links are to the respective comments – Adam’s and Teyo’s reply).

I’m going to quote Teyo’s comment in full because I think it answers a lot of question that people have had about some of the key differences between Puppet and Chef – dependency modelling and DSL:

There is a misstatement in your assessment of Puppet’s dependency handling. You express Chef’s ordering as deterministic and imply that Puppet is in someway non-deterministic. This is not the first time you have implied this publicly, so I thought I should bring some clarity to your misstatement. The actual differentiation is procedural ordering versus a dependency graph. Puppet provides a graphing model for ordering versus a procedural model. Sure, you get procedural ordering for free with Ruby, it seems easier, and I am aware that this was a design decision for you guys. We also know that you were frustrated by “having” to express dependencies in Puppet in order to ensure consistent ordering. Properly expressed dependencies in Puppet provide ordering where you care to have it. Procedural ordering is implicit even if you don’t care. This is a BIG difference, perhaps the fundamental difference between Puppet and Chef and one that was designed into Puppet because of experiences we had trying to cope with a large code base of procedurally order scripts to manage an enterprise infrastructure. Yeah we were using make, yeah that was crazy, crazy but informative.

Your omission is related to your design decision to avoid dependency graphing, which you yourself have admitted has some major downsides, namely the inability to provide a reasonable dry-run mode, http://bit.ly/4Gcz7G. Frankly, I don’t know how you develop with out a dry-run mode, but hey I am a sysadmin not a developer.

Without a graph of resource dependencies, we would have no way of separating concerns. Consider the use case of implementing security standards. Ideally, you would want any given configuration run to bring your system into complete compliance. That sounds great but would you really want security policies not to be implemented because some earlier procedure was unable to succeed, say because it was pulling in data from a source that was not available.

So here is the difference in a nutshell. Puppet generates a catalog of dependent resources. This catalog is shipped to the clients and acted on by the Resource Abstraction Layer (RAL). On the other hand Chef, ships the required Ruby code for any node’s configuration and orders the execution of that code procedurally. These are the core differences. The DSL issues become moot if you consider Shadow Puppet or the Ruby DSL that we are developing as part of Puppet’s next release. The real difference, and IMHO Puppet’s advantage is our resource model and it’s dependency graphs versus a monolithic procedural chunk of Ruby code delivered to every client.

Here are some derived advantages of our model and a little love for the much maligned declarative external DSL:

1) Graphing base branch independence.

Parts of a catalog can be implemented more often than others. That is to say, we can tag certain resources to be checked and reconfigured more often. Additionally, parts of a configuration can be meaningfully checked but not acted on (See Adam’s discussion of noop http://bit.ly/4Gcz7G). Our customers/community love this and without a graph I don’t see how it is possible.

2) Cross host dependencies.

Our data model passes dependencies into our catalog caching system, so cross host dependencies can be resolved as well. This isn’t currently available but the framework exists and we intend to take advantage of it.

3) Failures are contained.

Critical parts of a configuration run are not excluded because of non-dependent failures.

4) Low barrier to entry for non-rubyist.

Non-rubyist can take advantage of the specification language out of the gate and Ruby developers can take advantage of the current Puppet plugin API and the future Ruby DSL, so everyone gets to use their favorite hammer.

5) Don’t like our DSL, don’t like Ruby?

Because we are only generating a catalog from the configuration language it should be fairly straight forward for anyone to generate a catalog using whatever language they choose and the RAL would be able to act on it. Come on Python people you know you want to generate catalogs with Python.

6) I can’t run Ruby on my switch!

Because we are using a data model for resources, devices that can’t/don’t have access to Ruby could still use the catalog as a basis for configuring themselves. Routers, switches, firewalls, could all be configured using the same specification language independent of how the specification is implemented, but with the resource model intact.

Finally, I think that you were a little hard on John about his comments on Chef being Rails focused. Certainly he misspoke, but the truth is that Chef development has been focused on web-application rollout in fairly homogeneous environments. Sure you can use Chef to manage the initial deployment of a web application, but in environments where you may have lots of teams utilizing compute resources for various application architectures, Puppet’s resource model shines. Security administrators can develop their Puppet manifests and not need to worry that security policies are not going to be applied because the DBA teams manifests failed. Operations teams can run Puppet in noop mode persistently and be notified if their configuration is out of compliance. Developers can make sure that the infrastructure they need for successful application deployment is available without having to worry that the security policy failed to be applied. Everyone gets to be friendlier with one and other and perhaps even get to the pub earlier on occasion.

Disclosure & Disclaimer – I am Puppet’s Release Manager and obviously heavily involved with the Reductive Lab’s team and the project.  My opinions are my own and not representative of my employer or Reductive Labs.

Puppet 0.25.3 – “Clifford” released!

January 12th, 2010

CliffordPuppet 0.25.3 – code-named “Clifford”

The 0.25.3 release is a maintenance release in the 0.25.x branch.  The release addresses a regression introduced in 0.25.2 that caused issues with command execution.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.3.tar.gz

http://reductivelabs.com/downloads/puppet/puppet-0.25.3.gem

http://gemcutter.org/gems/puppet

Please note that all final releases of Puppet are signed with the Reductive Labs key.

http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying-puppet-downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.3.

CHANGELOG

* Bug #1464: Mount resource complains about missing options field
* Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #2887: Service (init) does not seem to work with require properly
* Bug #3013: util.rb:execute broken on Ruby <1.8.3
* Bug #3025: apt and aptitude providers dont work on Debian Lenny puppet 0.25.2 from gems

Puppet 0.25.2 “Zoe” released!

January 5th, 2010

Zoe the Muppet

Puppet 0.25.2 – code-named "Zoe"

The 0.25.2 release is a significant maintenance release (123 tickets closed!) in the 0.25.x branch.

Thanks to all who contributed to the release and tested fixes – especially (but not limited to!) Peter Meier (duritong), R.I. Pienaar (Volcane), Mark Plaskin, Dan Bode, Alan Harder, Ricky Zhou, Christian Hofstaedtler, Todd Zullinger, Till Mass, Nigel Kersten, and especially Markus Roberts and Jesse Wolfe who worked around the clock to get the release out the door.

The release is available at:

http://reductivelabs.com/downloads/puppet/puppet-0.25.2.tar.gz
http://reductivelabs.com/downloads/puppet/puppet-0.25.2.gem

Please note that all future final releases of Puppet will be signed with the Reductive Labs key.  Unfortunately, I am travelling and unable to access to the box with the release key on it or its backup.  A signature will be generated for this release early next week when I return to Australia.

http://reductivelabs.com/trac/puppet/wiki/DownloadingPuppet#verifying-puppet-downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.2.

RELEASE NOTES

* When setting aliases using the host type now use the host_alias attribute rather than alias.

* Puppet now has the "manage_internal_file_permissions" option which allows you to enable or disable Puppet management of internal files, for example those in /var/lib/puppet.  When "false" Puppet will NOT manage these files.  Default is "true".

* Cron type now supported on AIX

* Mailist type is now working again

* File serving permissions error messages enhanced

* SELinux now supports contexts with upper case titles

* When running the tests you no longer need to use RSpec version 1.2.2 but rather versions including and newer than.

* The debug format message has been changed and clarified from:

debug: Format s not supported for Puppet::FileServing::Metadata; has not implemented method 'from_s'

to:

debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson

* Puppetdoc now works with Regex node names

* There are now valid and proper OIDs in the LDAP puppet.schema that are unique and registered for Puppet.

* Packagers please note updated man pages including a new page for puppetqd

*    Fix for temporary file issues (https://bugzilla.redhat.com/show_bug.cgi?id=502881)

CHANGELOG

Full list of closed tickets.

 

Photobooth James

January 3rd, 2010

The Tortoise and not the Hare – Part 1

January 2nd, 2010

production line

The production line is one of the marvels of the industrial era. I've always been fascinated with production lines in factories and how a product, like a car, gets constructed from individual components and grows until finally it rolls off the production line as a finished product.  In the last few years I've been thinking more and more about production lines and how they overlap with IT Operations.  So do they have anything in common?  Can you draw parallels between building a car and running an IT operation? Damn right you can!

Production lines construct assets from components and then sells these assets.  IT shops also construct assets, in the form of software, infrastructure and services.  These assets are also constructed from components to make a functioning whole and are then sold to customers.  The perfect example is hosting infrastructure.  A host is constructed from hardware (CPU, storage, networking), software (operating system, applications) and configuration data and then delivered to a customer for use (or "sale"). I'm going to look at the core principles of production lines, specifically some of the methodologies around their management, and see if they offer value in running IT organisations and operations. I'm also going to demonstrate some work flow and how to use some tools (like Puppet and others) to model these principles in your own IT shop.

The production line relies heavily on process and continuous flow to function efficiently.  The asset moves through the line having actions performed on it or components added to it. The objective is an uninterrupted flow from beginning to end with enough of the right components, processes and people being introduced at the right time.  Getting this production life cycle right isn't easy. As a result, the study and practise of production line management has become a science.

One of the most famous methodologies is the Toyota Production System or TPS. If you work anywhere where process is important – and that's pretty much every manufacturing organisation and almost every large corporate (including banks, insurance companies, transport, logistics and a flurry of others) – then you'll probably have heard of TPS and one of its integral components, Kanban. The TPS is a lean/Just In Time (JIT) production practice model.  Lean practises are ones where the focus is on the activities that deliver customer value.  Resources that are expended on other activities are always suspect and targets for elimination.  JIT attempts to improve ROI ("Return on Investment") by streamlining production process, managing demand and hence reducing the amount of inventory ("parts") carried so that only the parts needed are stored and then for the shortest possible time before being consumed for production.

So where does Kanban fit in? Kanban is a demand management and signalling system that uses physical signs to act as triggers between processes.  I've stolen a very simple kanban example from Wikipedia:

"A simple example of the kanban system implementation might be a "three-bin system" for the supplied parts (where there is no in-house manufacturing) — one bin on the factory floor (demand point), one bin in the factory store and one bin at the suppliers' store. The bins usually have a removable card that contains the product details and other relevant information — the kanban card. When the bin on the factory floor becomes empty, i.e, there is demand for parts, the empty bin and kanban cards are returned to the factory store. The factory store then replaces the bin on the factory floor with a full bin, which also contains a kanban card. The factory store then contacts the supplier’s store and returns the now empty bin with its kanban card. The supplier's inbound product bin with its kanban card is then delivered into the factory store completing the final step to the system. Thus the process will never run out of product and could be described as a loop, providing the exact amount required, with only one spare so there will never be an issue of over-supply. This 'spare' bin allows for the uncertainty in supply, use and transport that are inherent in the system."

Simple huh?  You can also readily scale this example by adding multiple bins (which each have their own kanban card).  This allows tights controls on stock management (and hence costs!). 

So can we apply these concepts to IT infrastructure?  Hang onto your hats because in Part II of this series of posts I'm going to do exactly that.

Puppet 0.25.2 Release Candidate 3 out!

January 1st, 2010

We've pounced on a few more bugs and Puppet 0.25.2 release candidate 3 is a go.  Please test hard.  The production release should be in a few days barring any more bugs being found.