Posts Tagged ‘process’

The Tortoise and not the Hare 2 – Principles

January 24th, 2010

Kanban

In my first post I introduced you to the Toyota Production System and the Kanban signalling system. At the core of the TPS is the concept of maintaining efficiency and eliminating waste.  To govern this processes the TPS has a series of basic principles that articulate how this is achieved:

  1. Create continuous process flow to bring problems to the surface
  2. Use the “pull” system to avoid overproduction
  3. Level out or smooth the workload AKA “Heijunka” – be the tortoise not the hare
  4. Build a culture of stopping to fix problems, to get quality right from the first
  5. Standardized tasks are the foundation for continuous improvement and employee empowerment
  6. Use visual control so no problems are hidden
  7. Use only reliable, thoroughly tested technology that serves your people and processes.

I’ve skipped all the TPS rules around long-term thinking (for example the first principle of the TPS – Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals), corporate harmony, people development and organisational learning.  I haven’t skipped them because they aren’t important but because they aren’t immediately relevant to this discussion.  If you haven’t got the others right too though I suspect your organisation will go pear-shaped in other ways.

In each of the subsequent posts I am going to look at one of the seven principles I’ve articulated above, starting with exploring how you can make continuous process flow work for you.

The Tortoise and not the Hare – Part 1

January 2nd, 2010

production line

The production line is one of the marvels of the industrial era. I've always been fascinated with production lines in factories and how a product, like a car, gets constructed from individual components and grows until finally it rolls off the production line as a finished product.  In the last few years I've been thinking more and more about production lines and how they overlap with IT Operations.  So do they have anything in common?  Can you draw parallels between building a car and running an IT operation? Damn right you can!

Production lines construct assets from components and then sells these assets.  IT shops also construct assets, in the form of software, infrastructure and services.  These assets are also constructed from components to make a functioning whole and are then sold to customers.  The perfect example is hosting infrastructure.  A host is constructed from hardware (CPU, storage, networking), software (operating system, applications) and configuration data and then delivered to a customer for use (or "sale"). I'm going to look at the core principles of production lines, specifically some of the methodologies around their management, and see if they offer value in running IT organisations and operations. I'm also going to demonstrate some work flow and how to use some tools (like Puppet and others) to model these principles in your own IT shop.

The production line relies heavily on process and continuous flow to function efficiently.  The asset moves through the line having actions performed on it or components added to it. The objective is an uninterrupted flow from beginning to end with enough of the right components, processes and people being introduced at the right time.  Getting this production life cycle right isn't easy. As a result, the study and practise of production line management has become a science.

One of the most famous methodologies is the Toyota Production System or TPS. If you work anywhere where process is important – and that's pretty much every manufacturing organisation and almost every large corporate (including banks, insurance companies, transport, logistics and a flurry of others) – then you'll probably have heard of TPS and one of its integral components, Kanban. The TPS is a lean/Just In Time (JIT) production practice model.  Lean practises are ones where the focus is on the activities that deliver customer value.  Resources that are expended on other activities are always suspect and targets for elimination.  JIT attempts to improve ROI ("Return on Investment") by streamlining production process, managing demand and hence reducing the amount of inventory ("parts") carried so that only the parts needed are stored and then for the shortest possible time before being consumed for production.

So where does fit in? is a demand management and signalling system that uses physical signs to act as triggers between processes.  I've stolen a very simple example from Wikipedia:

"A simple example of the system implementation might be a "three-bin system" for the supplied parts (where there is no in-house manufacturing) — one bin on the factory floor (demand point), one bin in the factory store and one bin at the suppliers' store. The bins usually have a removable card that contains the product details and other relevant information — the card. When the bin on the factory floor becomes empty, i.e, there is demand for parts, the empty bin and cards are returned to the factory store. The factory store then replaces the bin on the factory floor with a full bin, which also contains a card. The factory store then contacts the supplier’s store and returns the now empty bin with its card. The supplier's inbound product bin with its card is then delivered into the factory store completing the final step to the system. Thus the process will never run out of product and could be described as a loop, providing the exact amount required, with only one spare so there will never be an issue of over-supply. This 'spare' bin allows for the uncertainty in supply, use and transport that are inherent in the system."

Simple huh?  You can also readily scale this example by adding multiple bins (which each have their own card).  This allows tights controls on stock management (and hence costs!). 

So can we apply these concepts to IT infrastructure?  Hang onto your hats because in Part II of this series of posts I'm going to do exactly that.

Google Chrome

September 13th, 2008

I downloaded Google’s new Chrome browser (http://www.google.com/chrome) this morning. I installed it and I played with it for about an hour. I read the notes and watched several of Google’s videos. Overall, it looks cool, seemed to be snappy and quick to respond. I was particularly taken with the focus on tabs. I also thought the multi-process, multi-tab sandbox idea is a really interesting idea – initially from a useability perspective but potentially also from a security perspective. Though there isn’t anywhere near enough information yet to make a proper assessment. I did try to break some tabs and see how effective the sand-boxing was (it seems to hold up from a very brief look but it’ll take some review from someone with more -fu than me to determine actually how secure the concept is – see below).

A few things didn’t inspire me – the border decoration was a little … unintegrated. And I am always loath to pass judgement on an application that by its very nature needs to be examined in a cross-platform context and I would want to see running on all of Microsoft Windows, OSX, and Linux. This is particularly true of OSX where the graphical environment can make an enormous difference to how an application engages you.

But after my play I closed Chrome down with a big sigh to get on with my actual day job. Why the sigh? Did I think the new Chrome browser wasn’t very good? Nope. But my first thought was “I wonder how many people in my enterprise have downloaded and installed Chrome over the last few days”. This was quickly followed by me asking two questions:

1. What is the change in the enterprise’s risk profile of adding this new application?

2. What’s the operational impact of some, many or all of my users downloading and installing this application?

Obviously (and hopefully) it is only an incremental change in risk profile and not much at that. The browser will probably only be downloaded by power users and innovators in the enterprise, initially at least. On the threat landscape in most enterprises however browsers punch well above their weight in terms of attack surface, are a common source of malware infection and browser exploits are a popular target for hackers. Indeed a (the first?) Chrome vulnerability has already been discovered AND exploited (see http://www.readwriteweb.com/archives/security_flaw_in_google_chrome.php) literally hours after Chrome was released. So Chrome’s potential as a source of compromise and attacks needs to be carefully considered.

The answer to the second question is also ambiguous but finding out can add a lot of work for a security team. With any new application, but especially ones like browsers that are such rich sources of malware attack, there is now a potential need to:

• Track bugs and vulnerabilities for the application;
• Add it to software profiles for vulnerability scanning;
• Investigate its behaviour for network behaviour and IDS/IPS;
• Profile it for our Security Event and Incident Management process; and
• Specifically for this application ascertain if its Incognito “stealth” browsing capability impacts our ability to investigate and gather evidence in incidents.

So most important to me right now is not whether Chrome will outshine Firefox or IE or whether it represents the future of the browser. But rather how much work is this new browser going to create for me… :)

P.S. If you’re interested in look at it you can find Chrome’s source at http://.google.com/chromium/. There are also build instructions (for which you will probably need to have some developer skills) for OSX and Linux.

Puppet’s BuildBot

August 24th, 2008

So rather than doing the work I actually should be I’ve been playing with BuildBot. I had intended to get around to setting up BuildBot sometime in the next couple of months but I got hooked.

The reason I wanted to have a look at BuildBot was that Puppet has reached a stage where we simply can’t test every platform it runs on. We are also starting to get patches from a wider variety of sources. Buildbot will allow us to execute our tests on a wider variety of platforms. Hopefully with the cooperation of the we can gather a really big collection of build platforms to test on.

Here’s the blurb for BuildBot

The BuildBot is a system to automate the compile/test cycle required by most software projects to validate changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.

The overall goal is to reduce tree breakage and provide a platform to run tests or -quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin.

It’s a very easy tool to deploy. The hardest part has been the slightly broken source handling and the assumption that any repository is local. I need to have a local repository to allow BuildBot to submit the right commits references to the PBChangeSource function.

But I designed a basic process for handling new commits:

1. Commit pushed to GitHub.
2. Commit bot at GitHub picks up commit and sends it to BuildBot Master.
3. BuildBot uses the git_buildbot.py script to calculate the before/after commit and branch references and tell BuildBot about them.
4. BuildBot executes the build and tells each slave to retrieve the commit and runs the tests. Currently we’re running:

a. All the Unit tests
b. All the RSpec tests

5. We then get the results of the tests on the website and in an email to the new Puppet Builds mailing list.

In addition I’ve also enabled BuildBot’s bot and added a new bot, called pinocchio, to the #puppet channel that reports on build status.

At this stage it’s all in test mode and when I’ve ironed out a few issues we should be in a position to do a production installation at ReductiveLabs and start canvassing for build slaves.

UPDATE

After mucking around with Buildbot I just couldn’t get a whole bunch of issues with resolved so we changed to as our – which works much better.  The message overall is – and : still a young pair.  I’ve included our configuration below for edification:

# -*- python -*-
# ex: set syntax=python:

# This is a sample buildmaster config file. It must be installed as
# ‘master.cfg’ in your buildmaster’s base directory (although the filename
# can be changed with the –basedir option to ‘mktap buildbot master’).

# It has one job: define a dictionary named BuildmasterConfig. This
# dictionary has a variety of keys to control different aspects of the
# buildmaster. They are documented in docs/config.xhtml .

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### BUILDSLAVES

# the ‘slaves’ list defines the set of allowable buildslaves. Each element is
# a tuple of bot-name and bot-password. These correspond to values given to
# the buildslave’s mktap invocation.
from buildbot.buildslave import BuildSlave

c['slaves'] = [BuildSlave("debian", "debian"),
BuildSlave("freebsd", "freebsd"),
BuildSlave("redhat", "redhat")
]

# ‘slavePortnum’ defines the TCP port to listen on. This must match the value
# configured into the buildslaves (with their –master option)

c['slavePortnum'] = 9989

####### CHANGESOURCES

# the ‘change_source’ setting tells the buildmaster how it should find out
# about source changes. Any class which implements IChangeSource can be
# put here: there are several in buildbot/changes/*.py to choose from.

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

####### SCHEDULERS

## configure the Schedulers

from buildbot import scheduler

stable = scheduler.Scheduler(name=”stable”, builderNames=["debian_stable", "freebsd_stable", "redhat_stable"], treeStableTimer=60, branch=”0.24.x”)
dev = scheduler.Scheduler(name=”dev”, builderNames=["debian_dev", "freebsd_dev", "redhat_dev"], treeStableTimer=60, branch=”master”)

c['schedulers'] = [stable, dev]

####### BUILDERS

# the ‘builders’ list defines the Builders. Each one is configured with a
# dictionary, using the following keys:
#  name (required): the name used to describe this bilder
#  slavename (required): which slave to use, must appear in c['bots']
#  builddir (required): which subdirectory to run the builder in
#  factory (required): a BuildFactory to define how the build is run
#  periodicBuildTime (optional): if set, force a build every N seconds

# buildbot/process/factory.py provides several BuildFactory classes you can
# start with, which implement build processes for common targets (GNU
# autoconf projects, CPAN perl modules, etc). The factory.BuildFactory is the
# base class, and is configured with a series of BuildSteps. When the build
# is run, the appropriate buildslave is told to execute each Step in turn.

# the first BuildStep is typically responsible for obtaining a copy of the
# sources. There are source-obtaining Steps in buildbot/steps/source.py for
# CVS, SVN, and others.

from buildbot.process import factory
from buildbot.steps import source, shell

pstable = factory.BuildFactory()
pstable.addStep(source.(repourl=’://github.com/jamtur01/puppet.’, branch=’0.24.x’))
pstable.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pstable.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

pdev = factory.BuildFactory()
pdev.addStep(source.(repourl=’://reductivelabs.com/puppet’, branch=’master’))
pdev.addStep(shell.ShellCommand(command=’rake spec’, name=’Spec Tests’))
pdev.addStep(shell.ShellCommand(command=’rake unit’, name=’Unit Tests’))

debian_stable = {‘name’: “debian_stable”,
‘slavename’: “debian”,
‘builddir’: “debian_stable”,
‘factory’: pstable,
}

debian_dev = { ‘name’: “debian_dev”,
‘slavename’: “debian”,
‘builddir’: “debian_dev”,
‘factory’: pdev,
}

redhat_stable = {‘name’: “redhat_stable”,
‘slavename’: “redhat”,
‘builddir’: “redhat_stable”,
‘factory’: pstable,
}

redhat_dev = { ‘name’: “redhat_dev”,
‘slavename’: “redhat”,
‘builddir’: “redhat_dev”,
‘factory’: pdev,
}

freebsd_stable = {‘name’: “freebsd_stable”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_stable”,
‘factory’: pstable,
}

freebsd_dev = { ‘name’: “freebsd_dev”,
‘slavename’: “freebsd”,
‘builddir’: “freebsd_dev”,
‘factory’: pdev,
}

c['builders'] = [debian_stable, debian_dev, freebsd_stable, freebsd_dev, redhat_stable, redhat_dev]

####### STATUS TARGETS

# ‘status’ is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and bots.

c['status'] = []

from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010))

from buildbot.status import mail
c['status'].append(mail.MailNotifier(fromaddr=”buildbot@reductivelabs.com”,
extraRecipients=["puppet-build@googlegroups.com"],
sendToInterestedUsers=False))

from buildbot.status import words
c['status'].append(words.(host=”.freenode.net”, nick=”pinocchio”,
channels=["#puppet"],
password=”password”))

# from buildbot.status import client
# c['status'].append(client.PBListener(9988))

####### DEBUGGING OPTIONS

# if you set ‘debugPassword’, then you can connect to the buildmaster with
# the diagnostic tool in contrib/debugclient.py . From this tool, you can
# manually force builds and inject changes, which may be useful for testing
# your buildmaster without actually commiting changes to your repository (or
# before you have a functioning ‘sources’ set up). The debug tool uses the
# same port number as the slaves do: ‘slavePortnum’.

#c['debugPassword'] = “debugpassword”

# if you set ‘manhole’, you can ssh into the buildmaster and get an
# interactive python shell, which may be useful for debugging buildbot
# internals. It is probably only useful for buildbot developers. You can also
# use an authorized_keys file, or plain telnet.
#from buildbot import manhole
#c['manhole'] = manhole.PasswordManhole(“tcp:9999:interface=127.0.0.1″,
#                                       “admin”, “password”)

####### PROJECT IDENTITY

# the ‘projectName’ string will be used to describe the project that this
# buildbot is working on. For example, it is used as the title of the
# waterfall HTML page. The ‘projectURL’ string will be used to provide a link
# from buildbot HTML pages to your project’s home page.

c['projectName'] = “Puppet”
c['projectURL'] = “http://reductivelabs.com/trac/puppet/”

# the ‘buildbotURL’ string should point to the location where the buildbot’s
# internal web server (usually the html.Waterfall page) is visible. This
# typically uses the port number set in the Waterfall ‘status’ entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.

#c['buildbotURL'] = “http://10.0.0.x:8010/”

Beer, beer. beer

May 16th, 2007

I’ve started home-brewing again and I am currently writing to the sound of a bubbling fermenter. Ruth asked me: “Is it supposed to make that noise.” Yes dear, it is. It’s called fermentation.

The current batch is a Summer Wheat beer and I have a MSB Two Row Lager finishing in bottles at the moment too. Temperature is a little dodgy I think for the finishing so we’ll have to see how we go. Certainly I’ve needed to pop a brew mat under the fermenter to keep the temperature up and get the fermentation going. But I can’t really think of a nice, warm’ish place to finish the beer at the moment.

I’m going to log batches in weblog entries. I’ve always been hopeless about documenting what and how I brew’ed batches. I should also record first and last gravity too but I keep forgetting to take measurements at the beginning. But hopefully having a location and a reminder might improve my process and documentation.

multipart/mixed: Power-User Email with the Sidekick

January 20th, 2007

This is a useful tip for Hiptop/Sidekick covering how to overcome the weaknesses of its email client. The Hiptop/Sidekick treats everything as a POP3 server (even when you set your email account’s server to IMAP or IMAP-SSL). This makes things a little tricky if you rely on IMAP as I do. So instead this tip suggests that you can do a couple of things to work-around some of the limitations.

Firstly, to allow you to sync email sent from your Hiptop with your primary email account you can:

1. Change the “From” and “Reply-To” addresses on your Hiptop to your primary email account.
2. Add a bcc from your Hiptop to your primary email account, every email sent from your Hiptop then gets also sent to your primary email address
3. Add a procmail (or sieve – see below) rule on your primary email server filing everything bcc’ed from your Hiptop into your Sent folder.

Secondly, to send all email from your primary email account you can add a procmail or sieve rule to cc all email directly to your email address. This is best done for non-mailing list emails and after spam processing has taken place.

The tip in the link above uses a procmail recipe for some of this but I prefer sieve and I’ve included that recipe below.


if address “return-path” “email@hiptop.com.au” {
addflag “\\Seen”;
fileinto “Sent”;
stop;
}

*insert whatever filing/anti-spam stuff in here*

if size :under 100K {
redirect “email@hiptop.com.au”;
keep;
}

keep;

CRAM-MD5 authentication with Dovecot IMAP

October 16th, 2006

I recently migrated my IMAP server from Courier-IMAP to Dovecot. It’s part of a whole simplification process I am engaging in. I cut-over the IMAP server and then last week enabled the Dovecot authentication in Postfix to allow me to stop using a separate SASL daemon for authentication. Now both SMTP and IMAP are authenticated from the one source – Dovecot.

But one thing I discovered when setting up Dovecot is that there is very limited documentation on using CRAM-MD5 authentication with Dovecot. As a result I am going to quickly document the process I used to get this up and running.

Firstly you need to enable the mechanism and specify a passwd database file in Dovecot. The mechanism and passdb file are specified in the dovecot.conf configuration file, on my system this is located in the /usr/local/etc/ directory.

auth default {
# Space separated list of wanted authentication mechanisms:
# plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi
mechanisms = plain login cram-md5

# passwd-like file with specified location
passdb passwd-file {
# Path for passwd-file
args = /etc/cram-md5.pwd
}

….

}

I’ve added the cram-md5 mechanism to the mechanisms statement and then added a passdb file, /etc/cram-md5.pwd.

Next, you need to create this passdb file and set appropriate permissions.

# touch /etc/cram-md5.pwd
# chmod 0600 /etc/cram-md5.pwd

After creating the file you need to add your users and hashed passwords to the passdb file. The users and passwords are added in the format:

user:passwordhash

Dovecot has a utility that allows you to convert passwords to the appropriate hashes. This utility is called dovecotpw and is installed into the /usr/local/sbin directory or is available in the source package in the src/util directory. You can run dovecotpw like so:

# dovecotpw
Enter new password:
Retype new password:
{HMAC-MD5}26b633ec8bf9dd526293c5897400bddeef9299fad

Enter the user’s password when prompted and it will be converted and outputted as a hash. The default hashed output is in the HMAC-MD5 scheme (which is appropriate for CRAM-MD5). You can change the scheme of the outputted hashes using the -s command line switch. Now add the generated password to the passdb file, /etc/cram-md5.pwd.

kartar:{HMAC-MD5}26b633ec8bf9dd526293c5897400bddeef9299fad

Finally, restart Dovecot and test authentication by enabling the appropriate mechanism in your email client. For example, to enable CRAM-MD5 authentication in Thunderbird you need to check the “Use secure authentication” checkbox in the Account Settings page.

Obviously I recommend that you use TLS/SSL to encrypt the authentication process as well.

A new spam strategy for me

September 27th, 2006

I get a lot of email. Mountains of the stuff. A fair chunk of it is spam. A consequence of having some older email addresses that still forward mail to me. As a result I spend quite a bit of time perfecting spam defenses. I used amavisd-new, dspam, grey-listing, SpamAssassin, SPF, DK/DKIM, p0f, bogofilter and Anomy and anti-virus all mingled together and tuned to an inch of their lives. My mail server burns some serious memory and CPU and has the world’s most damned complicated Postfix configuration. Recently, however, I threw my hands in the air and said “Enough!” Another approach was needed.

So I started doing some investigation and discovered qpmstpd – which is a very fast SMTP proxy written in Perl that comes with plug-ins for a variety of purposes – again all written in Perl. It sits in front of my Postfix and processes mail for spam and then submits it to the queue. After some initial tweaking it worked perfectly and the resulting configuration has greatly – greatly – simplified my mail configuration. The reduction in Postfix complexity alone (I nearly needed a flowchart to work out where mail was routed and injected) makes life considerably easier. The simplified anti-spam configuration in qpsmtpd is also much easier to manage.

Initial testing also reveals only a slight increase in spam slipping through and that’ll reduce as I tune SpamAssassin with some more training. Performance is also much improved and my box isn’t sucking up anywhere near as much memory and CPU – which might prolong the lifespan of the box.

I did need to add a new plug-in to do SMTP authentication for qpsmtpd. Luckily I found an older, not functional, plug-in that I adapted and fixed. You can find the updated auth_smtpd plug-in at the qpsmtpd wiki. Enjoy.

Omnigraffle versus Visio

April 28th, 2006

I draw/create a lot of diagrams – logical architectures, network diagrams, flowcharts, process workflows and a huge number of exposition diagrams designed to explain complicated concepts in visual form (I am currently particularly proud of my ISO17799 exploded framework diagram).

To create these diagrams I have become a Visio master. Now unlike most of the other Office products becoming a guru in Visio is a lot harder than it looks. Visio is Office’s little lost cousin. It was acquired in 2000 from Visio (formally Shareware and before that Axon and founded by a crew of ex-Aldus people) and thrown into the Business Products Division and then ‘integrated’ with Office. Whilst generally it behaves like a good Windows/Office application – uses the right shortcuts for Print/Save/Open etc – it also has quite a few quirks (bad habits) that date from its previous owners. It has also not always been the world’s most stable application. It is certainly not easy or intuitive to learn to use and trying to get it to do some seemingly basic things can be most troublesome. Indeed the general process of constructing diagrams can be time-consuming and painful. In summary, Visio mostly looks like other Office products, smells kinda like other Office products but occasionally leaves a bad taste in your mouth.

Then I bought a Powerbook and it came with a trial version of OmniGraffle – the Mac equivalent of Visio. Now I had no desire to learn a new product but last week on short notice I had to produce a logical architecture and only had my Powerbook with me. So I opened up OmniGraffle and started to work. Thirty minutes later I had my architecture. It was quick and easy and unlike almost every Visio diagram I’ve ever drawn it actually looked good without me needing to spend hours tuning colours and layout. I was quietly stunned. Simple, intuitive and generates great diagrams. I have since gone out and purchased a full license for the Pro version (which allows the import and export of Visio diagrams) and consider it money well spent. It sadly doesn’t run under Windows.

published

April 20th, 2006

Das Book II – Pro Nagios – has been released. So now its weeks of nerves about reviews and hoping people like it. It’s pre-selling very well at Amazon which is great. Seemingly better than the first book. But it’s very hard to get a feel for sales with only Amazon Sales Rank to go by (that’s even if you can work out Amazon Sales Rank and translate that into units moved).

Still don’t know what I want to do about a 2nd Edition of the first book. The thought of fixing and expanding the previous work is pretty exciting – I have a lot of ideas about how to go about that. But I am nervous about the time commitment. It’s an incredibly time consuming process – nearly a million words written over the last two years – and I need to work out whether I can achieve a better work/life balance and not end up a little nuts like I did last time. And I guess there were other factors involved in writing and putting in the time previously too. Guess some thinking and talking needs to be done.