This week, I answered a question about automated QA tools. This topic comes up frequently – my base answer is “use the simplest tool that will get the job done”. There are a lot of open-source tools, but I find that JUnit (or NUnit, if you’re in .NET) and Selenium or Watij are a great combination for testing web applications.
Read my answer here: http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1346325_tax306196,00.html
This week, I answered a question about writing a test plan, on my “Ask the Experts” column on searchsoftwarequality.com. You can read the answer at http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1346327_tax306121,00.html
What do you think? Do you use a test plan? If you’re agile, is your test plan this in-depth?
Back in October 2008, I answered the question "What's the Best Software Testing/QA Tool". In my answer, I included a few steps for ensuring you get an answer when asking questions.
Due to popular demand, I'm moving those comments into their own blog entry. This way, it's easier to find and read.
Summary? Be specific, research before you ask, solve problems, and give back. That's how to get answers online--in a sustainable fashion.
In my capacity as a software quality expert at http://searchsoftwarequality.com, I recently answered a question regarding software testing reports. A writer asked how to build a testing status report focusing on test execution criteria, test case failure rate, and bug rate.
You can see my answer posted at http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1345318,00.html
What do you think - too simplistic? Or do you agree that these are the bread-and-butter reports for software testing status?
A couple of days after the now infamous Zune disaster struck, the dogpile began. I don't mind critical bloggers, and I certainly don't mind customers complaining about product issues. As a software quality professional, I encourage both. But I tell... Esther Schindlers recent blog post on the Zune disaster really rubs me the wrong way!
Schindler's stance is that this is an obvious bug which everyone should have caught. That's simply wrong, and it's outrageously unprofessional to make that statement! Someone as experienced as she purports to be should have the courtesy to be honest in their analysis. We've been building software since, what, the sixties. Since 1970, there have been nine leap years, and there were four in the past eighteen years. What this means should be obvious: if this were a low-hanging, "obvious" bug, it would have popped up already. It would be on our common radar and we would be testing for it.
Another issue I take with Schindler's rant is that the developer who wrote this bug is 'still out there, in the wild, writing more bugs of the same nature'. Yeah, I will grant you that there are many developers writing stupid bugs like this. In my last organization, no less than three developers wrote the SAME bug in three different releases (it was a URL information disclosure/escalation bug). It was a silly bug and should have never been written once, let alone thrice. But in almost thirteen years at Microsoft, I've never seen a engineer last who has made repeat mistakes, period. Fool me once, shame on me. Fool me twice, this way is the door! Her concerns that a similar bug will be lurking in Microsoft's next software are just hype to sell columns. There's little or no substance to them.
She also complains that the bug is simple date math. I think she's jumping to inexperienced conclusions here--first of all, I see little in her resume to lead me to believe she's an active developer qualified to typecast a bug as a date math defect. Secondly, all we know is that, due to being the last day in a leap year, the Zune froze. One driver from one component used in one Zune release froze. Is it date math, or something else? Only the Zune team really knows and it would be presumptuous for anyone else to claim they knew the cause.
So one lesson, she says the lesson is that it's a failure of the development and QA process. Uh, duh. No kidding! Second lesson she learned is that we need to learn from history (of course, there's no real history to learn from here except that people make mistakes). Third lesson is that the developer is still working. Uh, duh... If we fired every developer who wrote a bug, we'd just have testers left looking at their hands! If we were to lose our jobs over mistakes, we'd all be unemployed! I'm not sure exactly what Schindler's looking for here, but I can tell you I don't like the direction she's going.
Schindler's post is mindless crowd-mongering, aimed at stirring up a crowd (and keeping her name in print). It's also rife with wrong conclusions and, if her assertions were taken to heart, the result would be drastically unrealistic.
But let's ask the question: should Microsoft, and the engineering community within and without, be worried about this defect? Yes! What are the real take-aways from this issue? Read on...
The first take-away here is that this is a bug with high potential repeatability. While this is the first known instance of this defect, it does seem possible that it could happen again. So lesson learned - update the test repository. Next time you test leap year handling, be sure to cover not just Feb 28, 29, March 1, 2 but also December 31.
More than updating the repository with this specific test case, testers need to extrapolate a series of test cases. There's something happening here. Schindler might, in fact be right that it's a math bug. It might also just be a buffer overrun (365 days were used up by Dec 30) or something else. The point is, there's a class of bug to be researched and investigated. It's around date management, date accounting, infrequent and semi-regular change and similar topics. And you can bet that both the developer and the feature tester on this component is working on this!
Bill Gates' vision for Trustworthy Computing has always included the concept of distributed programming, where the application model is heavily object-oriented and componentized. The mistake many testers and organizations make is they only want to perform system or functional testing on the completed product. One of the key lessons of Agile and XP development methodologies is that you break things up into pieces and test them. According to the press release, this bug is at the component level and that's where it should be caught.
Organizations need to reconsider their risk-based approach, and ask if component-level testing (unit testing, functional testing: different names for the same activity) would be more effective. Component-level testing, combined with rapid feedback (ie, daily builds with testing involved) will prevent defects from creeping in at the component level, and might have caught an issue like this one.
This third point is a personal mission for me. In the goal to minimize investment, IT organizations are constantly squeezing test. Development can overrun dates willy-nilly, but it's always a requirement for the test org to 'make it up. Hey, it's business and we need to hit deadlines. But applying the same effort at the development level (in this case, the effort of increasing discipline to stay on schedule) as the test team is expected to apply (in this case, the effort of catching up when dev has blown the unrealistic schedule) could prevent the need for herculean test effort, and would yield more time for the test organization to think about and improve upon strategy.
Why does this matter? Finding defects like this requires creativity. It requires looking beyond functional test cases and thinking about 'what could go wrong'. Schindler's making the same mistake many people make - overlooking the fact that testers have to catch all of the bugs, whereas the user only needs to experience one. Yes, we sign up for this when we enter a career in quality, but that doesn't change the fact that there's a disparity in expectations. A highly successful, high-quality release cannot be achieved if you force-march your test team and cut corners.
Most IT organizations get away with this approach because their customers are internal and, therefore, are more forgiving. But this kind of bug is very similar to security defects. They're difficult to find, require a lot of creativity, and take time to root out.
So, a great big raspberry to Esther Schindler for a poorly-written article on the Zune incident. True, customers were served poorly. More importantly, the engineering community needs to take lessons here. Her lessons? I question the logic used in drawing her conclusions and I really feel her posting was a populist knee-jerk lacking any substantive value for the engineering community.
At the same time, there is plenty to be learned, and learn we better! Let's build off this new class of defect, let's approach our testing from a more component perspective, and let's think about how much time testing is given to being creative.
Full disclosure: yes, I work at Microsoft as a Senior SDET Lead. But I'm critical of my employer where it makes sense. If I were the manager of the Zune test org (and I know several people in the org), I'd be having the same conversation with them that I wrote in this blog posting.
In my role as a volunteer youth leader in my Church, I have the opportunity to help put together a monthly activity. My group was responsible for the activity this month, and we chose to offer five team building exercises (all straight from http://www.wilderdom.com/games/InitiativeGames.html). We did four exercises in a rotation, and then all met together to perform the fifth. The exercises were:
What is interesting to me is the way teams interacted to get the problem solved. As I said, I conducted the helium stick challenge. As teams started, they were astonished that their stick ROSE instead of sunk. In a flash, right on the heels of that recognition, came frustration. We were split up into groups of boys and girls--oddly enough, while both groups expressed frustration, only the boys started yelling at each other. I kid you not, they were really ripping on one another. They quickly moved into the blame game. Each group (all four) needed to be stopped multiple times.
After frustration/blame, the groups moved into 'try harder' mode. I finally would stop each group and point out that trying harder wasn't working - maybe there was a better way. At that point, it was again quite funny - no single group picked an organized way to brainstorm. It was 'herd brainstorming' and they all just started shouting ideas.
Eventually someone 'won' through all the shouting. In one group, it was an adult leader, who started back into try harder. In a group of 12-year old boys, one of the boys got everyone's attention and he literally talked the stick to the ground. You need to understand the magnitude of that feat - you've got eight people all pushing up on a stick (pushing up because they have to keep their fingers on the stick at all times, and by doing so they force the stick up). Somehow this boy talked everyone, step by step, through the process of lowering the stick.
The girls were pretty creative--they caught on quickly that they needed to coordinate, and touch was the best way to do so. One group of older girls interlocked pinky fingers and lowered together. Another group split into two smaller groups on each end, and just put their hands touching, next to one another.
For me the takeaway was:
How can this apply to software development, software testing and software quality? Well, software is built by teams - even the smallest unit of engineering is a feature team made of two or more people. Communication is important, and it's critical that 1) no one resorts to blame and 2) each person is allowed to share their point of view. "Writing unit tests just so we can hit a goal of coverage seems to be distracting us from the real goal--our output is actually lower quality right now than it used to be" needs to be answered with "Why do you think that way?" and not "Uh huh - not true. Besides, you're not helping at all with all these bugs you're bothering me with!"
Good communication includes illustrating the current status, as in "Hey wait, everyone seems to be pushing up" and then a group discussion of what's causing that: "We're pushing up because we all want to keep our finger on the stick." Only then can the group move on and start thinking about the solution. So in an engineering organization, that might be "why are we getting a flood of bugs? Wait, testers are off doing something else (building a battleship test automation system) rather than focusing on testing daily builds". Once the situation is recognized, only then can the team react with an appropriate response.
As quality assurance/software testing teams work with and communicate with development counterparts, quality becomes a natural by-product. As teams discuss what practices are producing current results, they can move forward and improve how they approach the challenge of writing quality software. A little communication can move our engineering teams from try harder to work smarter, and output increases in both quantity and quality.
