Archive for January, 2010

Puppet 0.25.4 released!

January 29th, 2010

You wanted “release early, release often” and the team has delivered!
The 0.25.4 release a maintenance release (with one important feature – pre/post transaction hooks – discussed below) in the 0.25.x branch.  The release primarily addresses a regression introduced in 0.25.3 that caused issues with creating cron jobs.

The release available at:

http://reductivelabs.com/downloads//-0.25.4.tar.gz

http://reductivelabs.com/downloads/gems/-0.25.4.gem

http://gemcutter.org/gems/

Please note that all final releases of are signed with the Reductive Labs key – http://reductivelabs.com/trac//wiki/DownloadingPuppet#verifying…

Please report feedback via the Reductive Labs Redmine site: http://projects.reductivelabs.com

Please select an affected version of 0.25.4

RELEASE NOTES

Pre/Post Transaction hooks

There a new feature in this release: pre and post transaction hooks.  These hooks allow you to specify commands that should be run pre and post a configuration transaction.   They are set with the prerun_command and postrun_command settings in the .conf configuration file.

prerun_command = /bin/runbeforetransaction
postrun_command = /bin/runaftertransaction

The command must exit with 0, i.e. succeed, otherwise the transaction will fail – if the pre command fails before the transaction run and if the post command fails at the end of the transaction.

CHANGELOG
*  Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #3001: Can’t manage broken links
* Bug #3039: 0.25.3 gem spec specifies the executables incorrectly
* Bug #3075: sshkey host aliases broken by fix for #2813
* Bug #3088: Puppetd fails to stop after receiving SIGTERM
* Bug #3089: puppetlast gsub! error
* Bug #3093: Blastwave provider broken in 0.25.3
* Bug #3104: Test failed: ::Network::XMLRPCClient when performing the rpc call and an exception   raised.should log and raise XMLRPCClientError if Timeout::Error raised
* Bug #3112: Problem with adding and removing crons
* Bug #3122: Uncharacterized failure in fileserving under OS X
* Bug #3125: Dpkg tests failing
* Feature #2914: Transactions should have before and after hooks

The Tortoise and not the Hare 2 – Principles

January 24th, 2010

Kanban

In my first post I introduced you to the Toyota Production System and the Kanban signalling system. At the core of the TPS the concept of maintaining efficiency and eliminating waste.  To govern this processes the TPS has a series of basic principles that articulate how this achieved:

  1. Create continuous flow to bring problems to the surface
  2. Use the “pull” system to avoid overproduction
  3. Level out or smooth the workload AKA “Heijunka” – be the tortoise not the hare
  4. Build a culture of stopping to fix problems, to get quality right from the first
  5. Standardized tasks are the foundation for continuous improvement and employee empowerment
  6. Use visual control so no problems are hidden
  7. Use only reliable, thoroughly tested technology that serves your people and processes.

I’ve skipped all the TPS rules around long-term thinking (for example the first principle of the TPS – Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals), corporate harmony, people development and organisational learning.  I haven’t skipped them because they aren’t important but because they aren’t immediately relevant to this discussion.  If you haven’t got the others right too though I suspect your organisation will go pear-shaped in other ways.

In each of the subsequent posts I am going to look at one of the seven principles I’ve articulated above, starting with exploring how you can make continuous flow work for you.

Puppet, Chef, deterministic ordering and the much maligned DSL

January 14th, 2010

This morning I came across a post entitled Puppet versus Chef: 10 reasons why Puppet wins.  The post attempts to explain the differences between Chef and and why superior.  The post wasn’t great IMHO, personally I thought it was fairly poorly reasoned and made some, potentially accurate, but throughly unsubstantiated claims.

Leaving aside the issues with the post itself though, it did prompt an interesting comment thread, particularly comments  between Opcode’s CTO Adam Jacob and Reductive Lab’s Teyo Tyree (links are to the respective comments – Adam’s and Teyo’s reply).

I’m going to quote Teyo’s comment in full because I think it answers a lot of question that people have had about some of the key differences between and Chef – dependency modelling and :

There a misstatement in your assessment of ’s dependency handling. You express Chef’s ordering as deterministic and imply that in someway non-deterministic. This not the first time you have implied this publicly, so I thought I should bring some clarity to your misstatement. The actual differentiation procedural ordering versus a dependency graph. provides a graphing model for ordering versus a procedural model. Sure, you get procedural ordering for free with Ruby, it seems easier, and I am aware that this was a design decision for you guys. We also know that you were frustrated by “having” to express dependencies in in order to ensure consistent ordering. Properly expressed dependencies in provide ordering where you care to have it. Procedural ordering implicit even if you don’t care. This a BIG difference, perhaps the fundamental difference between and Chef and one that was designed into because of experiences we had trying to cope with a large code base of procedurally order scripts to manage an enterprise infrastructure. Yeah we were using make, yeah that was crazy, crazy but informative.

Your omission related to your design decision to avoid dependency graphing, which you yourself have admitted has some major downsides, namely the inability to provide a reasonable dry-run mode, http://bit.ly/4Gcz7G. Frankly, I don’t know how you develop with out a dry-run mode, but hey I am a sysadmin not a developer.

Without a graph of resource dependencies, we would have no way of separating concerns. Consider the use case of implementing security standards. Ideally, you would want any given configuration run to bring your system into complete compliance. That sounds great but would you really want security policies not to be implemented because some earlier procedure was unable to succeed, say because it was pulling in data from a source that was not available.

So here the difference in a nutshell. generates a catalog of dependent resources. This catalog shipped to the clients and acted on by the Resource Abstraction Layer (RAL). On the other hand Chef, ships the required Ruby code for any node’s configuration and orders the execution of that code procedurally. These are the core differences. The issues become moot if you consider Shadow or the Ruby that we are developing as part of ’s next release. The real difference, and IMHO ’s advantage our resource model and it’s dependency graphs versus a monolithic procedural chunk of Ruby code delivered to every client.

Here are some derived advantages of our model and a little love for the much maligned declarative external :

1) Graphing base branch independence.

Parts of a catalog can be implemented more often than others. That to say, we can tag certain resources to be checked and reconfigured more often. Additionally, parts of a configuration can be meaningfully checked but not acted on (See Adam’s discussion of noop http://bit.ly/4Gcz7G). Our customers/community love this and without a graph I don’t see how it possible.

2) Cross host dependencies.

Our data model passes dependencies into our catalog caching system, so cross host dependencies can be resolved as well. This isn’t currently available but the framework exists and we intend to take advantage of it.

3) Failures are contained.

Critical parts of a configuration run are not excluded because of non-dependent failures.

4) Low barrier to entry for non-rubyist.

Non-rubyist can take advantage of the specification language out of the gate and Ruby developers can take advantage of the current plugin API and the future Ruby , so everyone gets to use their favorite hammer.

5) Don’t like our , don’t like Ruby?

Because we are only generating a catalog from the configuration language it should be fairly straight forward for anyone to generate a catalog using whatever language they choose and the RAL would be able to act on it. Come on Python people you know you want to generate catalogs with Python.

6) I can’t run Ruby on my switch!

Because we are using a data model for resources, devices that can’t/don’t have access to Ruby could still use the catalog as a basis for configuring themselves. Routers, switches, firewalls, could all be configured using the same specification language independent of how the specification implemented, but with the resource model intact.

Finally, I think that you were a little hard on John about his comments on Chef being Rails focused. Certainly he misspoke, but the truth that Chef development has been focused on web-application rollout in fairly homogeneous environments. Sure you can use Chef to manage the initial deployment of a web application, but in environments where you may have lots of teams utilizing compute resources for various application architectures, ’s resource model shines. Security administrators can develop their manifests and not need to worry that security policies are not going to be applied because the DBA teams manifests failed. Operations teams can run in noop mode persistently and be notified if their configuration out of compliance. Developers can make sure that the infrastructure they need for successful application deployment available without having to worry that the security policy failed to be applied. Everyone gets to be friendlier with one and other and perhaps even get to the pub earlier on occasion.

Disclosure & Disclaimer – I am ’s Release Manager and obviously heavily involved with the Reductive Lab’s team and the project.  My opinions are my own and not representative of my employer or Reductive Labs.

Puppet 0.25.3 – “Clifford” released!

January 12th, 2010

Clifford 0.25.3 – code-named “Clifford”

The 0.25.3 release a maintenance release in the 0.25.x branch.  The release addresses a regression introduced in 0.25.2 that caused issues with command execution.

The release available at:

http://reductivelabs.com/downloads//-0.25.3.tar.gz

http://reductivelabs.com/downloads//-0.25.3.gem

http://gemcutter.org/gems/

Please note that all final releases of are signed with the Reductive Labs key.

http://reductivelabs.com/trac//wiki/DownloadingPuppet#verifying--downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.3.

CHANGELOG

* Bug #1464: Mount resource complains about missing options field
* Bug #2845: Cron entries using “special” parameter lose their title when changed
* Bug #2887: Service (init) does not seem to work with require properly
* Bug #3013: util.rb:execute broken on Ruby <1.8.3
* Bug #3025: apt and aptitude providers dont work on Debian Lenny 0.25.2 from gems

Puppet 0.25.2 “Zoe” released!

January 5th, 2010

Zoe the Muppet

0.25.2 – code-named "Zoe"

The 0.25.2 release a significant maintenance release (123 tickets closed!) in the 0.25.x branch.

Thanks to all who contributed to the release and tested fixes – especially (but not limited to!) Peter Meier (duritong), R.I. Pienaar (Volcane), Mark Plaskin, Dan Bode, Alan Harder, Ricky Zhou, Christian Hofstaedtler, Todd Zullinger, Till Mass, Nigel Kersten, and especially Markus Roberts and Jesse Wolfe who worked around the clock to get the release out the door.

The release available at:

http://reductivelabs.com/downloads//-0.25.2.tar.gz
http://reductivelabs.com/downloads//-0.25.2.gem

Please note that all future final releases of will be signed with the Reductive Labs key.  Unfortunately, I am travelling and unable to access to the box with the release key on it or its backup.  A signature will be generated for this release early next week when I return to Australia.

http://reductivelabs.com/trac//wiki/DownloadingPuppet#verifying--downloads

Please report feedback via the Reductive Labs Redmine site:

http://projects.reductivelabs.com

Please select an affected version of 0.25.2.

RELEASE NOTES

* When setting aliases using the host type now use the host_alias attribute rather than alias.

* now has the "manage_internal_file_permissions" option which allows you to enable or disable management of internal files, for example those in /var/lib/.  When "false" will NOT manage these files.  Default "true".

* Cron type now supported on AIX

* Mailist type now working again

* File serving permissions error messages enhanced

* SELinux now supports contexts with upper case titles

* When running the tests you no longer need to use RSpec version 1.2.2 but rather versions including and newer than.

* The debug format message has been changed and clarified from:

debug: Format s not supported for ::FileServing::Metadata; has not implemented method 'from_s'

to:

debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson

* Puppetdoc now works with Regex node names

* There are now valid and proper OIDs in the LDAP .schema that are unique and registered for .

* Packagers please note updated man pages including a new page for puppetqd

*    Fix for temporary file issues (https://bugzilla.redhat.com/show_bug.cgi?id=502881)

CHANGELOG

Full list of closed tickets.

 

Photobooth James

January 3rd, 2010

The Tortoise and not the Hare – Part 1

January 2nd, 2010

production line

The production line one of the marvels of the industrial era. I've always been fascinated with production lines in factories and how a product, like a car, gets constructed from individual components and grows until finally it rolls off the production line as a finished product.  In the last few years I've been thinking more and more about production lines and how they overlap with IT Operations.  So do they have anything in common?  Can you draw parallels between building a car and running an IT operation? Damn right you can!

Production lines construct assets from components and then sells these assets.  IT shops also construct assets, in the form of software, infrastructure and services.  These assets are also constructed from components to make a functioning whole and are then sold to customers.  The perfect example hosting infrastructure.  A host constructed from hardware (CPU, storage, networking), software (operating system, applications) and configuration data and then delivered to a customer for use (or "sale"). I'm going to look at the core principles of production lines, specifically some of the methodologies around their management, and see if they offer value in running IT organisations and operations. I'm also going to demonstrate some work flow and how to use some tools (like and others) to model these principles in your own IT shop.

The production line relies heavily on and continuous flow to function efficiently.  The asset moves through the line having actions performed on it or components added to it. The objective an uninterrupted flow from beginning to end with enough of the right components, processes and people being introduced at the right time.  Getting this production life cycle right isn't easy. As a result, the study and practise of production line management has become a science.

One of the most famous methodologies the Toyota Production System or TPS. If you work anywhere where important – and that's pretty much every manufacturing organisation and almost every large corporate (including banks, insurance companies, transport, logistics and a flurry of others) – then you'll probably have heard of TPS and one of its integral components, Kanban. The TPS a lean/Just In Time (JIT) production practice model.  Lean practises are ones where the focus on the activities that deliver customer value.  Resources that are expended on other activities are always suspect and targets for elimination.  JIT attempts to improve ROI ("Return on Investment") by streamlining production , managing demand and hence reducing the amount of inventory ("parts") carried so that only the parts needed are stored and then for the shortest possible time before being consumed for production.

So where does fit in? a demand management and signalling system that uses physical signs to act as triggers between processes.  I've stolen a very simple example from Wikipedia:

"A simple example of the system implementation might be a "three-bin system" for the supplied parts (where there no in-house manufacturing) — one bin on the factory floor (demand point), one bin in the factory store and one bin at the suppliers' store. The bins usually have a removable card that contains the product details and other relevant information — the card. When the bin on the factory floor becomes empty, i.e, there demand for parts, the empty bin and cards are returned to the factory store. The factory store then replaces the bin on the factory floor with a full bin, which also contains a card. The factory store then contacts the supplier’s store and returns the now empty bin with its card. The supplier's inbound product bin with its card then delivered into the factory store completing the final step to the system. Thus the will never run out of product and could be described as a loop, providing the exact amount required, with only one spare so there will never be an issue of over-supply. This 'spare' bin allows for the uncertainty in supply, use and transport that are inherent in the system."

Simple huh?  You can also readily scale this example by adding multiple bins (which each have their own card).  This allows tights controls on stock management (and hence costs!). 

So can we apply these concepts to IT infrastructure?  Hang onto your hats because in Part II of this series of posts I'm going to do exactly that.

Puppet 0.25.2 Release Candidate 3 out!

January 1st, 2010

We've pounced on a few more bugs and 0.25.2 release candidate 3 a go.  Please test hard.  The production release should be in a few days barring any more bugs being found.