Departing from my quality-focused blog postings for a while (software testing rules, but hey – sometimes there’s good info elsewhere too), here’s a pointer to something you need to read!
John Glazer is a CIO and contributor to HIS talk (THE blog for healthcare IT). He captures the essence of a healthy IT org so well, it’s a must-read. And it is quick!
This is a blitz blog – quick, to the point, and hopefully helpful. Topic: persistence.
My wife has a cat (a HUGE cat, although my aspiring veterinarian son tells us she’s a ragamuffin breed, which is ‘big boned’ by nature). This cat I do not like. Don’t know why, but we’ve never really gotten along. But lately, she’s really been trying. She comes into my office, right next to my desk, lays down, rolls over, and just purrs. All the while, she looks right at me with the kindest, happiest look on her face. Finally, I give up and pet her.
Can we learn from the big cat? When we want something, can we be pleasantly persistent, just waiting patiently for the attention we’re looking for? I can look back on my career and see many times where some pleasant persistence might have paid off better than being a bit more forceful).
I’ve been reading “Fearless Change” and have found the message in this book to be the same persistent patience that our cat is demonstrating.
I frequently read a daily thought on families and how to improve your family. Yesterday’s thought dovetailed neatly into something thinking I had on QA organizations: are you a thermostat, or a thermometer?
A thermostat is a device which regulates the temperature in your home. It controls the atmosphere. A thermometer is a device which reflects the temperature. To be a thermometer is to be reactive; all it does is indicate the temperature. To be a thermostat means to proactively set the temperature.
In the family, a ‘thermostat’ parent doesn’t react to grumpiness or other negative emotions, but calmly sets the tone in spite of what’s going on around her or him. A ‘thermometer’ parent, on the other hand, gets ‘set off’ easily when children bicker, whine, or complain. Obviously my goal as a husband and father is to be a thermostat, although there are ‘thermometer-like’ days!
In a recent thread on the Yahoo Agile Testing Group (about continuous release), I noticed more than one poster saying something like “I tell management about an issue, and let them decide.” This reactive approach to QA has been a pet-peeve of mine. It smacks of cop-out to me, and I’ve seen it develop into a victim mentality among many testers—myself included.
In my tenure at Microsoft (where I’m ‘serving’ my second stint, with a combined tenure exceeding 12 years), the testing organization has been consistently active in pushing for quality. A few things differentiate Microsoft from other organizations: first, the engineers wear the pants (that’s changing, slowly). In fact, nothing ships if the program manager, development manager, and test manager sign off on it. And they won’t sign off unless their teams sign off. Secondly, any individual has the right to stop a project, even the newest tester in an organization. Third, while the management structure is massively deep (I’m some six or seven layers below Steve Ballmer, current CEO and President), any individual can get the ear of their VP, Senior VP, and CEO.
So the culture I’m most familiar with is one wherein testing is an active participant in product development. The testing organization moves beyond a ‘show and tell’ relationship with management to an active participant in business strategy. The testing organization is part of thermostat, and rarely plays a thermometer role.
I have worked in organizations where this is not the case, and I’ve seen a lot of evidence that this often isn’t the case. In many IT and product software organizations, testing plays a thermometer role. We ‘take the temperature’ of a piece of software, and let management decide when to ship. In fact I have been told by management, when I’ve tried to be a thermostat, to back off and let them make the decisions. In one case, I have even refused to sign off on a risky release (security-related issue on a piece of beta software), telling the release organization they were welcome to release but they’d have to convince my manager to sign off because I wasn’t going to.
Thermostats regulate the temperature. In my “Quality 2.0” vision, testing organizations participate in regulating quality. The test organization moves from just taking the project’s temperature, to a role where they are helping to set quality. To do this, testing teams need to:
These are just a few things a test organization can do to transition from a thermometer to a thermostat. What’s missing? Does your test team take a more proactive role somewhere else? Let’s hear it!
So in my new job, one of my first tasks is to put together an automation system--by this I mean a harness and a framework. The process has had me thinking (and talking) a lot about what makes a good system in general.
The automation harness is the system used to schedule, distribute and run tests and to record results. In the open source world, some tools used here include NUnit, JUnit, and TestNG (my personal favorite). These tools all work in a one-off situation - they are all run locally out of the dev environment or via the command line. In software testing at Microsoft, though, a one-by-one approach to automation is useful for 1) developer unit testing, 2) tester unit testing/test creation, and 3) failure investigation. However for the 24/7 test environment we're building, this isn't sufficient. We need a centralized scheduling tool that allows us to push tests out to multiple clients (simulate load and de-serialize test runs). So we're working internally to Microsoft, evaluating the existing automation harnesses available and trying to find the one that works best.
A big factor for me in this selection is finding a harness which is configurable. At Microsoft, we are pushing the envelope in testing in a variety of ways: code coverage analysis, failure analysis, and several similar activities which allow us streamline our testing, reduce test overhead and automate many testing tasks. This means our framework MUST be extensible - we have to be able to plug in new testing activities.
A second element of the automation system is the framework. This is the abstraction layer which separates our automated tests from the application under test. This abstraction layer is critical to good automation. If, for instance, you are automating a web application, you will probably experience a lot of churn in the application layout. You do not want your automated tests hard-coded looking for certain controls in certain locations (ie, in the DHTML structure)--by abstracting this logic, your test can call myPage.LoginButton.Click(), and your abstraction layer can 'translate' this into clicking a button located in a specific div. In some organizations, this framework is purchased. At the LDS Church, we leveraged both Selenium RC and Watij to build this framework, developing most of it ourselves internally (kudos to Brandon Nicholls and Jeremy Stowell for the work they did in this capacity).
The challenge felt by most test organizations is two-fold: 1) finding the engineering talent to build these systems and 2) making the investment in innovation. Ironically, the very thing which can free up resources for other tasks (automated testing) is the thing most managers don't want to spend time on! This makes sense, sort of... managers don't like to invest in activities which (in their mind) don't contribute directly to the bottom line. In all but the smallest of projects, this makes no sense--test automation isn't a sellable product, but if automated tests can free up a day or two of test time, that's a day or two spent doing other activities! Each and every time automation is being run.
Recruiting top talent is also a challenge. In both IT organizations where I worked, there was a culture among developers that testers weren't engineers--they were click testers. Testers couldn't give input on 'extremely technical concepts' like architecture, potential bug causes, or the likes--they're there to pick up code as developers release it, and then to find bugs. It's no wonder that it's so challenging to hire engineers into testing - when they're treated like that, they're either going to leave or move to development!
So the keys to a great automation system are: 1) a solid, extensible and flexible harness, 2) a robust framework, generally customized to your test activities, 3) management commitment to invest in innovation and automation and 4) top engineering talent and the culture to reward them for their contribution.
Am I missing anything?
40 IT failures caused by software bugs by ZDNet's Michael Krigsman -- Rick Hower, who runs the Software QA Test Resource Center has compiled a lengthy listing of “major computer system failures caused by software bugs” Here are several entries from that list: A September 2006 news report indicated problems with software utilized in a state government’s primary election, resulting in periodic unexpected rebooting of voter check-in machines, [...]This article is a great source of justification. I've had my own negative experiences caused by inadequate testing (I keep harping on it, but 13 million corrupted rows in a production database is a bad thing - before my time, so I don't bear responsibility, but I failed in several related arguments about the level of testing needed).
