Thoughts on QA and Engineering

How Difficult is it, Really?

2010-01-29T14:29:00.001-08:00

I’ve been using an easier version of my coding question lately, because candidates have struggled so much. First I moved it from C to C# (or Java, if candidates only have a java background). Then I simplified it, a lot. The problem is simple: reverse a string using no outside libraries. This was an introductory-level when I interviewed at Microsoft but now it seems to be advanced!

I’ve had candidates waste 40, 50 lines of code answering this. They usually don’t answer the question at all (they won’t write the full function with signature, or they reverse any words w/o reversing the characters, etc.).

I tried it again last night against .NET 3.x—I wanted to make sure I wasn’t expecting too much--you know, maybe programming has changed while I’ve been in meetings! ;).

I ended up with a small challenge (converting my char[] so it returned a string—kept returning the object type), but other than that, this went really well. 15 lines of code, including curly braces and the step to convert from the input string into a char[] (no error handling; have to get to that next)

Is the question too difficult? Wanna take a stab? Can you beat 15 lines, without using string.Reverse()?

Running MSTest Standalone

2009-12-18T07:44:00.001-08:00

It’s been a very long time since I posted on this blog – I’ve been quite busy with work as well as writing articles for http://www.searchsoftwarequality.com (head over and check out my tips and Expert Answers).

I stumbled across this today—how-to guide to running MS Test standalone. http://www.shunra.com/shunrablog/index.php/2009/04/23/running-mstest-without-visual-studio/

Full disclosure, during my last stint at Microsoft, I worked on this internally. However, I never disclosed how to do it. I do not know Shunra and I honor my non-disclosure to the fullest extent.

At the same time, I’m thrilled. MSTest is a great product and I’d really love it if MS produced a standalone version. According to this post by an MSFT employee, there’s a lot of discussion around it:

this is one of the things on a higher priorities that we are considering for a future release though this is yet being worked out as such there if no formal statement. I am pushing for this & may have a update in a couple months. http://social.msdn.microsoft.com/forums/en-US/vststest/thread/2897fb68-ef48-4941-a49e-fe8cb1b5aced/

So who knows, maybe we’ll see something soon!

Blitz Blog: Quality Assurance Through Code Analysis

2009-04-23T20:54:00.001-07:00

Today I read through a Parasoft whitepaper published on searchsoftwarequality.com, and I found it to be a great approach to static code analysis.

FULL DISCLOSURE: I write articles and am a “Software Testing Expert” for searchsoftwarequality.com. However, I do not blog positively if I don’t believe in the posting.

Parasoft is the manufacturer of an application security static code analysis tool. Of course, the white paper’s intention is to sell copies of their tool. Can’t fault them for 1) believing in their product and 2) wanting to pay the bills.

What I really like about this article is the approach they take to SCA. They are not pushing it as the end-all, be-all of code quality. They are very realistic about SCA, in fact, stating that it’s often over-used and, once over-used, ignored. Companies implementing SCA must be careful about it—don’t enable a rule unless you really want to enforce it.

At the same time, they make a strong point about the value of SCA. It can really help a team drive quality upstream. Some policies an engineering team might want to use, for example, contribute to code readability while other contribute to security (dynamically-built SQL statements vs. parameterized queries anyone?).

As an Agile engineer, I do take issue with their heavy-handed ‘management enforcement’ method. Agile teams need to adopt policies as needed. Cross-company Agile team representatives might establish company-wide policies and enforcement, but the Agile team itself should arrive at the bulk of the policy definitions.

One thing I like to see is the implementation of SCA directly in the build process – ie, no build if the code fails with errors, and a team-wide email on warnings. This is the best way to enforce policies (but teams need to be selective about policies, so they don’t overwhelm or ‘over-stay their welcome’).

Anyhow, blitz blog. Highly recommended reading!

http://go.techtarget.com/r/6673725/7930283/1

Technorati Tags: quality assurance,software testing,qa,security

Oracle Owns the Stack

2009-04-21T20:24:00.001-07:00

You’ve probably already heard that Oracle has acquired Sun Microsystems. It’s one big, happy, expensive enterprise systems family! Personally, I think this is not a good thing—it hampers competition. Oracle and Java already have a stranglehold on the IT space, and this merger just tightens the binds.

I have worked a total of 12 years at Microsoft, with two years in IT at other companies, which were both very big Oracle companies. I worked at Circuit City in the Oracle Retail implementation, and I worked at the LDS Church where most everything is running in Java on AIX. With my two years of experience, after working in Microsoft technologies, I have decided Oracle is overpriced, overhyped software. Java I could take or leave, but I firmly believe Oracle sells too little for the money.

Here’s the deal… Oracle sells their hardware and their software at astronomical prices. I mean, seriously… The hardware is expensive. The OS can often be open source, but it still requires people to manage it. If you add up the hardware, DB and other proprietary software, and the management software you’ve sunk a ton more money into than if you’d bought from the ‘Evil Empire’.

On top of that, I do not believe Oracle sells decent products. SQL is OK, but Oracle Retail? It’s a mish-mash of third-party acquisitions, none of which really share a common language or data format. And they bundle it all together and say “It’s Oracle – it’s enterprise quality” and people believe it. As a QA manager, I was appalled. For instance, I could NOT get the Oracle account rep on-site for Circuit to tell me what level of security testing the product had undergone. He told me that was proprietary information which he could not share. So we were investing millions in his software, and we didn’t have the right to know if it was secure? Another example: we tried for weeks (I paid three IBM consultants nearly $600 an hour for six weeks) to automate one of Oracle Retail’s components. The entire time, the Oracle rep was in our status meetings listening to us report blocked, unsuccessful, struggling… Never once did he peep up, until I finally asked how they automate regression testing. At first, he tried to evade with the same “that’s proprietary information” response. When he finally realized I wasn’t going to drop the issue, he admitted they didn’t automate anything in that area. What?? What kind of company watches a customer throw thousands (tens of thousands) of dollars down the drain, all the while knowing the effort is futile?

In my role as QA Engineer at the LDS Church, I was surprised to hear people tell me they would not deploy Microsoft products in the public-facing network segment because Microsoft controls the entire stack, and a security exploit in one layer of the stack meant the entire stack was at risk. ??? From a technology point of view, the opinion makes no sense. Ironically, these were the same people who approved of and supported internal development of security technology, when Microsoft had an existing product on the market (and deployed at the LDS Church). With Oracle now owning Sun, will that answer have to change?

So in a recent guest posting on HISTalk, Orland Portale (former GM, Global Health Industry) spouts on and on about how good this acquisition is. To me the nail in the coffin is the statement that Oracle will begin creating “tightly bundled system stacks which incorporate hardware and software components. Oracle will now have all layers of the systems stack under its umbrella, including the storage, server, operating system, programming language, database, Web services, etc.”

In recent years, Oracle has been snapping up enterprise solutions left and right. They own the entire retail suite. They bought BEA, one of the last independent web providers. They recently purchased MySQL and have control of it. At what point will all the “Evil Empire” opponents realize there’s a newer, much larger empire out there – and it controls all the software they deployed because they wanted to stay away from Microsoft technologies?

I know… Long rant about this. I admit I’m a Microsoft bigot. Except for one product (SMS, or System Center Cofiguration Manager), every enterprise server product I have deployed has just, well, worked! And been reliable and stable. So my bigotry is based on my personal experience. It doesn’t take a PhD to deploy MS software, it works well together, and it’s got great uptime. And the price is approachable, too.

Soon I think it’ll become obvious: Microsoft is the underdog here. Fight back, stop the Evil Empire. Buy Microsoft!

Tenants of a Software Testing Team

2009-04-16T20:54:00.001-07:00

If you had the chance to write out the tenants (unbreakable ground rules) by which your quality assurance team lives, what would they be? Here are some I’ve been thinking of:

We partner actively in quality-first development, by participating in the entire SDLC and by contributing tools, process, and time to the development activity.
We do not and cannot ‘test in quality’. We test to validate requirements, discover missing or inaccurate requirements and implementation, and to expose defects.
We use tools and process to discover defects, validate functionality, and improve quality. Not to use tools and process.
Secure software is a top priority—we protect our users’ privacy as well as our applications reliability.

What about you? Do you have other tenants on your team?

Technorati Tags: quality assurance,qa,software testing,engineering,leadership

Blitz Blog: John Glaser on the Healthy IT Org

2009-04-13T20:54:00.001-07:00

Departing from my quality-focused blog postings for a while (software testing rules, but hey – sometimes there’s good info elsewhere too), here’s a pointer to something you need to read!

John Glazer is a CIO and contributor to HIS talk (THE blog for healthcare IT). He captures the essence of a healthy IT org so well, it’s a must-read. And it is quick!

http://histalk2.com/2009/04/13/being-john-glaser-41409/

Technorati Tags: quality assurance,qa,testing,software testing

Blitz Blog: Persistent Quality Assurance

2009-04-09T20:23:00.001-07:00

This is a blitz blog – quick, to the point, and hopefully helpful. Topic: persistence.

My wife has a cat (a HUGE cat, although my aspiring veterinarian son tells us she’s a ragamuffin breed, which is ‘big boned’ by nature). This cat I do not like. Don’t know why, but we’ve never really gotten along. But lately, she’s really been trying. She comes into my office, right next to my desk, lays down, rolls over, and just purrs. All the while, she looks right at me with the kindest, happiest look on her face. Finally, I give up and pet her.

Can we learn from the big cat? When we want something, can we be pleasantly persistent, just waiting patiently for the attention we’re looking for? I can look back on my career and see many times where some pleasant persistence might have paid off better than being a bit more forceful).

I’ve been reading “Fearless Change” and have found the message in this book to be the same persistent patience that our cat is demonstrating.

Technorati Tags: quality assurance,qa,testing,software testing,engineering

Software Testing: Is Your Quality Assurance group a Thermostat, or a Thermometer?

2009-04-08T04:12:00.001-07:00

I frequently read a daily thought on families and how to improve your family. Yesterday’s thought dovetailed neatly into something thinking I had on QA organizations: are you a thermostat, or a thermometer?

A thermostat is a device which regulates the temperature in your home. It controls the atmosphere. A thermometer is a device which reflects the temperature. To be a thermometer is to be reactive; all it does is indicate the temperature. To be a thermostat means to proactively set the temperature.

In the family, a ‘thermostat’ parent doesn’t react to grumpiness or other negative emotions, but calmly sets the tone in spite of what’s going on around her or him. A ‘thermometer’ parent, on the other hand, gets ‘set off’ easily when children bicker, whine, or complain. Obviously my goal as a husband and father is to be a thermostat, although there are ‘thermometer-like’ days!

In a recent thread on the Yahoo Agile Testing Group (about continuous release), I noticed more than one poster saying something like “I tell management about an issue, and let them decide.” This reactive approach to QA has been a pet-peeve of mine. It smacks of cop-out to me, and I’ve seen it develop into a victim mentality among many testers—myself included.

In my tenure at Microsoft (where I’m ‘serving’ my second stint, with a combined tenure exceeding 12 years), the testing organization has been consistently active in pushing for quality. A few things differentiate Microsoft from other organizations: first, the engineers wear the pants (that’s changing, slowly). In fact, nothing ships if the program manager, development manager, and test manager sign off on it. And they won’t sign off unless their teams sign off. Secondly, any individual has the right to stop a project, even the newest tester in an organization. Third, while the management structure is massively deep (I’m some six or seven layers below Steve Ballmer, current CEO and President), any individual can get the ear of their VP, Senior VP, and CEO.

So the culture I’m most familiar with is one wherein testing is an active participant in product development. The testing organization moves beyond a ‘show and tell’ relationship with management to an active participant in business strategy. The testing organization is part of thermostat, and rarely plays a thermometer role.

I have worked in organizations where this is not the case, and I’ve seen a lot of evidence that this often isn’t the case. In many IT and product software organizations, testing plays a thermometer role. We ‘take the temperature’ of a piece of software, and let management decide when to ship. In fact I have been told by management, when I’ve tried to be a thermostat, to back off and let them make the decisions. In one case, I have even refused to sign off on a risky release (security-related issue on a piece of beta software), telling the release organization they were welcome to release but they’d have to convince my manager to sign off because I wasn’t going to.

The Testing Thermostat

Thermostats regulate the temperature. In my “Quality 2.0” vision, testing organizations participate in regulating quality. The test organization moves from just taking the project’s temperature, to a role where they are helping to set quality. To do this, testing teams need to:

Include experienced, respected engineers. At one time in my career, I worked with Christian Hargraves, probably one of the most experienced engineers I’ve ever met. Christian led and maintains the Jameleon project. He’s a sought after expert in the Salt Lake IT market, and has worked for the LDS Church as what I would call a Test Architect. Every developer in the organization knew Christian knew his stuff, and if Christian recommended something, that had weight. Qualified, respected engineers who are part of a test organization can help contribute to quality process and practices across the organization. They can influence core development teams to push quality into their process, from code reviews to adopting best practices.
Advocate actively about projects. Testers need to be active participants in release decisions, influencing senior management. In many instances, testers are shareholders. In every instance, testers share the benefits of a good release by enjoying continued employment, and in ‘worst case’ scenarios, testers go down in flames along with the rest of the company. In other words, our employment is just as dependent on a good release as any CEO’s. If all we do is report quality, without influencing decision, we are placing our career in the hands of other people. I do not accept this point of view!
Drive quality upstream: we all know it’s significantly cheaper to fix a bug in the planning phase, early implementation, or even release validation phases as opposed to post-release. Testers need to work to push themselves and their influence upstream—the earlier we can look at code, the better.
Be involved in planning activities. Take an active role in scheduling. Drive feedback about previous releases (especially lessons learned) into current project planning. If the company has a history of underestimating test duration, speak up about it. Track and present data which shows a more realistic approach. Work patiently but consistently to drive reality into schedules.
Hold dependencies accountable. If the development team isn’t dropping code early enough for full testing, don’t stand for that! Your reputation as a tester, not to mention your continued employment, depends on a solid release. If the company runs with firm release dates, but development is dumping code in your lap a week before release, you’re going to end up working 90+ hours that final week, and still releasing crappy code! Negotiate and set milestones, and hold development accountable for reaching those milestones. Two things will result: first, your test organization will start to work a consistent, sane amount of hours. Secondly, quality WILL go up because you’ll be looking at code earlier.
Be involved in establishing release criteria. This is critical—across your business, you need to have consensus on what constitutes release-readiness. If you leave this up to marketing, they’ll say a set of customer stories need to be developed. Leave it to development and they’ll say that, plus possibly that all unit tests need to pass. As QA engineers, we have a unique perspective in that we understand the value of positive AND negative testing. We can help influence the establishing of release criteria which truly reflect product quality. Setting this criteria early, well before a project is nearing a release deadline, removes the emotion and sets a rational basis for declaring ‘Ship Ready’.

These are just a few things a test organization can do to transition from a thermometer to a thermostat. What’s missing? Does your test team take a more proactive role somewhere else? Let’s hear it!

Technorati Tags: quality assurance,qa,software testing,quality

Found a Security Flaw (The Need for Software Testing Everywhere)

2009-04-03T14:15:00.001-07:00

Today I was paying a bill online, and I crashed the company’s IIS server (I promise, I did nothing wrong—intentionally). The good news: they write their code rather securely, so they are using parameterized queries rather than embedded SQL statements. The bad news (besides the crash itself) is that they were running with tracing enabled, so I saw the entire stack trace.

I called the company and got in touch with their tech guy. He was really polite on the phone and very open to feedback. I shared a bunch of info with him and thought I ought to document it. So here’s an Open Letter to All Admins Running IIS:

· You need to check your machine.config (and possibly web.config) files and make sure <trace enabled=”false” localOnly=”true” pageOutput=”false” requestLimit=”10” traceMode=”SortByTime”/> See http://msdn.microsoft.com/en-us/library/aa302351.aspx for more info.

· Either spin up Microsoft Baseline Security Analyzer (MBSA) or use IISLockDown and URLScan to scan your server(s) security, especially in configuration. See http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=f32921af-9dbe-4dce-889e-ecf997eb18e9

· You might also benefit from a security review/audit from an experienced, independent consultant.

Security Focus has a decent article on the issue: http://www.securityfocus.com/infocus/1755 (it’s old but still accurate). NOTE there is a chance that running IISlockdown may break something. If your developers built the site with some dependency on what’s actually a security hole, locking down your server could cause issues. I highly recommend that you run this against a test deployment and then run all of your automated tests against that deployment, and that you do some manual testing.

You might also consider picking up copies of

“Improving Web Application Security” (MS Press),
“Building Secure Microsoft ASP.NET Applications (MS Press),
“Writing Secure Code” (I *highly* recommend this – it’s the bible on secure coding, especially in MS technologies), and
“Secure Development Lifecycle” (MS Press).

From more to less specific, these books are pointers on secure IIS configuration, ASP.NET coding, coding in general, and strategies for maintaining application security throughout lifecyles. The latter book is a great read if you’re on a larger team and/or need to influence management to give you the time for security-related work.

Technorati Tags: software testing,testing,software security

DRAFT: What’s the Role of a Test Manager in Agile Organizations?

2009-02-12T20:23:00.001-08:00

This blog posting is a draft. It represents a work in progress, and is posted with the hope of gaining peer review for improvement.

Many people have asked me—or challenged me—about the role of a test manager in agile organizations. A basic premise of agile being the self-lead team, who needs a test manager? Many agile engineers even bristle at the thought.

Reality shows, however, that not all teams or organizations benefit from a complete lack of management. Smaller engineering organizations can often do without, but larger organizations perform better with some amount of centralized management. A team of ten engineers, with no role specialization, has no need for a test manager. A team of fifty engineers, which maintains a differentiation between test-oriented engineers and development-oriented engineers for a valid business reason will certainly benefit from a test manager (and, incidentally, a development manager).

What do I think is the role of a test manager in such an organization? I think the test manager contributes in five higher-order functions: interfacing with senior management; providing vision and leadership; fostering cross-group collaboration; leading specialties, tools and reporting; and providing organization management. That’s a lot of Dilbert-esque lingo! What do I really mean? Read on for details…

Interface With Senior Managers

Larger organizations, such as the team I led at Circuit City, have deep management structures. It’s simply a fact of life. And those managers generally manage cross-discipline organizations and benefit from help with a manager between them and each discipline. In Circuit’s MST project, I managed over one hundred engineers, with thirteen concurrent workstreams. My role was to interface between my manager and the test organization, communicating messages and centralized decisions in a way that addressed my teams’ needs, interests, and concerns.

Test managers also act as a buffer, insulating the team from senior management’s see-saw decision-making process. Sometimes just shielding testers from the questions senior management can ask is performing a huge favor—questions like “do we have the right number of testers” can really cause concern in a test team; the test manager can research and answer that question often without churning up any worried or troubled feelings.

When senior managers arrive at a strategic plan, the test manager’s job includes integrating that plan into the test organization. A great example would be an organization which has decided to migrate from waterfall to agile! The test manager’s role will change dramatically, moving from ‘manager’ to ‘coach,’ assisting the test team in understanding their role and helping to set engineering goals in terms of process adoption.

Another role test managers play in this larger organization is simply pushing back on bad ideas. Individual contributors are often heads-down (especially on agile projects), in the day-to-day work. If senior management stumbles upon a ridiculous idea, individual team members often lack the cycles (and sometimes the experience) to push back on that decision. The role of a test manager often includes helping senior management understand the possible side effects of a decision.

In some organizations, testing or quality assurance are not often viewed as critical. These organizations may lack experience, or may have experience in smaller projects than the one at hand. A test manager’s role includes the responsibility to advocate for quality across the entire company. Sometimes this means helping customers understand the value of the ‘extra cost’ of a few test engineers on a project. Other times, it’s helping executive management see the need for experienced testers. Sometimes the job includes preventing releases from happening when core quality issues beyond simple functionality could cost the company in terms of money or bad PR.

Provide Vision and Leadership

Agile teams are (rightfully) quite busy in the day-to-day and may not be cognizant of everything going on in the company. The test manager’s responsibility in an agile organization can include driving vision. With an understanding of company growth plans, the test manager can be looking ahead to understand how to double or triple the size of the test organization. The test manager can also be thinking about strategic redirection – for instance, perhaps the test organization depends on developers for all automated testing. The test manager can set in place organization-wide plans to train testers, helping them become better automation engineers. The test manager needs to be the person asking “Why not?” and helping to drive the team forward toward that goal.

Sometimes agile teams fall behind due to circumstances beyond their control. The agile approach is to take those hits, learn, and move on. But sometimes there’s value in getting help. For instance, if a member of an agile team becomes ill during a critical period, a strong test manager can step in and ‘pinch hit’ for that tester while he or she is out. This isn’t advocating that the test manager is simply a ‘reserve’ resource. Teams which underestimate or over-extend themselves need the experience which comes from missing goals. But there are times when a test manager can contribute at an individual level.

Another leadership role is to help teams through transition. The change from waterfall to agile is a fantastic example of this – as a leader, the test manager needs to play a critical, positive role in helping testers work through the difficult migration process. The test manager provides structure and understanding, offers insight, and encourages team members as they face challenges. Above all, the test manager needs to be supportive and reassuring during the struggles which come with change. This leadership will help retain top talent, which might otherwise leave the organization due to the uncertainty and stress of change.

Finally, the test manager can represent the test organization and the company as a whole. A good test manager can be a key asset in customer retention. By contacting external customers, being aware of situations they’re facing, and communicating the status of fixes and changes, the test manager can help the customer through any rough spots in project implementation. Another role the test manager can play is sitting as a member of industry or standards organizations, making sure quality has a voice at that table and providing expertise in the subject.

Foster Cross-Group Collaboration

A critical role played by any management in an Agile project is eliminating road blocks and keeping the team moving. The test manager can play a pivotal part in this by interacting with other groups, with the business side of projects, and with the customer. Sometimes it takes a management-level title to ‘influence,’ even within the same company. More important, some negotiations and even simply driving closure can involve a lot of time, talking, and walking. By remaining proactively involved in projects, the test manager can help remove impediments, all the while allowing the team to remain heads down.

Also, very often in a larger organization, the test manager is the first level team member with corporate purchasing privileges. Therefore, the test manager can often take care of licensing and other purchases and fees for supplies the team needs to carry on their work.

In larger engineering organizations, the test manager works closely with development and program manager/project manager counterparts. They collaborate on cross-discipline objectives as well as on budgets and scheduling. They are often the point-people for driving mitigation for lessons learned which surfaced in retrospectives.

Another role the test manager plays in a larger organization is introducing engineers around the company. A successful test manager is very familiar with projects and personnel across the entire organization; when one employee is looking for help, that test manager can make appropriate introductions.

Finally, the test manager can play a scrum master role in a scrum of scrums, helping to lead the meeting.

Specialties, Tools, and Reporting

Generally test organizations are split into teams in a matrixed structure. Most testers fall easily into a project team, where they’ll be busy until that project completes. There are, however, occasions where testers are hired who act in more of a ‘Center of Excellence” function. For instance, performance testing, automation and tools, or security-oriented testers. These testers are shared across the organization, brought in as experts to play a specific role on a release. Organizationally it’s often most efficient for these testers to report to the test manager, and the test manager manages their assignments.

In a large Agile organization, there is often tension between the Agile maxim “do what works'” and organizational efforts to standardize, especially on tools. Standardization sometimes aids in keeping costs down (when expensive commercial tools are used), especially for tools like load testing tools. Standardization also aids in maintaining resource ‘portability’ (the ability for a tester to move from team to team, without needing to learn new tools). A test manager needs to be very sensitive to the needs (perceived and real) of a team, but can often be instrumental in maintaining consistent toolsets across the organization.

Agile is notable, among other things, for a distinct lack of metrics. Status reporting is as important to senior management as it is a burden to agile teams. The test manager can play a critical role in pulling together status out of several test teams, presenting it to management in a format they’re used to. It seems a bit mundane, but in spite of the gains made moving to agile, many senior managers still yearn for detailed status reports.

Related to reports is the idea of common metrics. Just as the Agile team needs to ‘do what works,’ the Agile organization also needs to do what works. Sometimes that means a bit of a burden on the Agile team. With engineering expertise as well as an understanding of project management, a test manager can help Agile teams arrive at metrics which are easily generated (often times in an automated manner, if unit testing or defect tracking systems are in place). There are metrics that work – they vary from company to company, but they work—and the test manager is a key player in defining these metrics and automating the gathering process.

Finally, the test manager can offload the ‘compliance’ burden from the Agile team, reporting up to senior management on the progress against various company initiatives. Again, maybe not exciting to the test manager, but critical to the company’s improved effectiveness and efficiency (well, we like to hope each initiative is critical).

Organizational Leadership

Even in the Agile organization, things like performance management have to happen. The test manager carries a significant administrative burden driving this process. This is a good thing – a good test manager not only helps his or her teammates grow in skills and experience, they also advocate with senior management for promotion and compensation increases.

As a test manager, my experience has always been that my teams are more effective when I screen candidate for open positions, and then pull team members in on formal interview loops. As a test manager, I have had responsibility for opening job postings, authoring job descriptions, and sourcing resumes for open roles. When the test manager takes on this administrative burden, it allows the Agile team to focus on their work. When decent candidates are sourced, the team can stand down briefly to perform interviews.

Closely related to recruiting is the role the test manager takes on in hiring and onboarding new hires. The manager shuffles new hires through the hiring process, helps in signing up for benefits and such, and makes sure the new hire settles in well. A key role I have seen too many test managers overlook is introducing the new hire around the company. Finally, the new hire will achieve their highest potential as quickly as possible only if and as the manager assists them in setting goals for impact, development, and growth.

A test manager’s work is to drive his or her team to higher levels of effectiveness and efficiency. Working in concert with members of the Agile teams, test managers can accomplish this in part by researching and procuring appropriate training. Whether it’s bringing in experts in Agile testing or arranging for on-site training in automation skills, training is critical for improving what a test organization can accomplish. Procuring that training can be a time-consuming task. Also the test manager arranges for training which fits broad, cross-organization needs rather than focused on specific teams.

Finally, the test manager is critical in helping with employee retention. A good manager is aware of the interests and professional goals of each employee in the organization, and makes the effort to provide that employee with opportunities in sync with the employees wishes. With a broad, cross-organizational focus, the manager can also help employees find new roles when, without that assistance, the employee might have left the company.

Help the Agile Process Work

A closing concept for the test manager’s role in Agile is the idea that he or she is a make-or-break participant in the transition to Agile. It can take several months or even a year to successfully transition to an Agile process, and that transition can be painful—especially for testers! The test manager helps coach teams through the transition, aiding and lending support and encouragement, fostering courage, and assisting in power transformation. This is not a tops-down role, but a one-on-one, coaching role helping team members with courage, patience, and confidence.

Technorati Tags: quality assurance,qa,testing,software testing,agile

Quality Assurance: automated QA software testing tools

2009-01-31T08:48:00.001-08:00

This week, I answered a question about automated QA tools. This topic comes up frequently – my base answer is “use the simplest tool that will get the job done”. There are a lot of open-source tools, but I find that JUnit (or NUnit, if you’re in .NET) and Selenium or Watij are a great combination for testing web applications.

Read my answer here: http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1346325_tax306196,00.html

Technorati Tags: software testing,testing,software quality assurance,test automation

Software Testing: Writing a Test Plan

2009-01-31T08:44:00.001-08:00

This week, I answered a question about writing a test plan, on my “Ask the Experts” column on searchsoftwarequality.com. You can read the answer at http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1346327_tax306121,00.html

What do you think? Do you use a test plan? If you’re agile, is your test plan this in-depth?

Technorati Tags: software quality assurance,quality assurance,software testing,testing

How to Ask For Help with Software Testing/Quality Assurance Questions

2009-01-28T20:12:00.001-08:00

Back in October 2008, I answered the question "What's the Best Software Testing/QA Tool". In my answer, I included a few steps for ensuring you get an answer when asking questions.

Due to popular demand, I'm moving those comments into their own blog entry. This way, it's easier to find and read.

To ensure you get an answer:

To all testers: look before you ask. Really! If you are wondering what tool you can use for a given test type, use Live Search and look for information! Performing a minimal amount of research shows respect to the audience you're turning to for help, and can actually prevent a question now and again. QA is all about searching for product defects; apply that searching capability to answering your questions.
You're more likely to get help on a specific question rather than a broad question. For instance, asking "What's the right tool" won't get you much. However "I've evaluated Selenium, Selenium RC, and Watij--given my situation (describe it) what tool do you recommend?"
Talk about the problem you're trying to solve. So "We realized we'll be running tests over and over and over again, but our UI changes frequently. What strategies can we take...?" is a question that raises the problem and let's people know what help you're looking for.
Asking "what's the best tool?" is like asking "What's the best car?". If you live in Germany and drive the autobahn frequently, the best car for you is far different from the best car for someone who lives in Bombay, battles thick traffic, and drives Indian roads (notoriously rough). Be specific, give details, and look for recommendations. Software testing tools are the same way - a screaming "BMW" test tool will fall apart on a Bombay road. A right-hand drive Jaguar would be totally out of place on an American highway. A performance test tool isn't the right way to automate repeated quality assurance tests. The right tool is dependent on the project, the skill sets on the QA team, the timeline, and several other factors.
Solve your own problems. The Internet is an incredible tool and offers us all a ton of opportunity to not have to reinvent the wheel. But don't ask other people to do your work for you! Ask for advice. If you want someone to solve your problem, ask one of us to consult (for pay) and bring us in. We'll be glad to get the job done for you!
Give back: as you grow and learn, give back... Don't post a question, get your answer, and disappear. Remain an active participant in the community and 'pay it forward'.

Summary? Be specific, research before you ask, solve problems, and give back. That's how to get answers online--in a sustainable fashion.

Technorati Tags: software,testing,quality assurance,qa,software testing

Quality Assurance: software testing reports

2009-01-18T12:19:00.001-08:00

In my capacity as a software quality expert at http://searchsoftwarequality.com, I recently answered a question regarding software testing reports. A writer asked how to build a testing status report focusing on test execution criteria, test case failure rate, and bug rate.

You can see my answer posted at http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1345318,00.html

What do you think - too simplistic? Or do you agree that these are the bread-and-butter reports for software testing status?

Technorati Tags: quality assurance,qa,software testing,testing

The three REAL software testing/software quality lessons learned from the Zune incident

2009-01-06T20:18:00.001-08:00

A couple of days after the now infamous Zune disaster struck, the dogpile began. I don't mind critical bloggers, and I certainly don't mind customers complaining about product issues. As a software quality professional, I encourage both. But I tell... Esther Schindlers recent blog post on the Zune disaster really rubs me the wrong way!

Schindler's stance is that this is an obvious bug which everyone should have caught. That's simply wrong, and it's outrageously unprofessional to make that statement! Someone as experienced as she purports to be should have the courtesy to be honest in their analysis. We've been building software since, what, the sixties. Since 1970, there have been nine leap years, and there were four in the past eighteen years. What this means should be obvious: if this were a low-hanging, "obvious" bug, it would have popped up already. It would be on our common radar and we would be testing for it.

Another issue I take with Schindler's rant is that the developer who wrote this bug is 'still out there, in the wild, writing more bugs of the same nature'. Yeah, I will grant you that there are many developers writing stupid bugs like this. In my last organization, no less than three developers wrote the SAME bug in three different releases (it was a URL information disclosure/escalation bug). It was a silly bug and should have never been written once, let alone thrice. But in almost thirteen years at Microsoft, I've never seen a engineer last who has made repeat mistakes, period. Fool me once, shame on me. Fool me twice, this way is the door! Her concerns that a similar bug will be lurking in Microsoft's next software are just hype to sell columns. There's little or no substance to them.

She also complains that the bug is simple date math. I think she's jumping to inexperienced conclusions here--first of all, I see little in her resume to lead me to believe she's an active developer qualified to typecast a bug as a date math defect. Secondly, all we know is that, due to being the last day in a leap year, the Zune froze. One driver from one component used in one Zune release froze. Is it date math, or something else? Only the Zune team really knows and it would be presumptuous for anyone else to claim they knew the cause.

So one lesson, she says the lesson is that it's a failure of the development and QA process. Uh, duh. No kidding! Second lesson she learned is that we need to learn from history (of course, there's no real history to learn from here except that people make mistakes). Third lesson is that the developer is still working. Uh, duh... If we fired every developer who wrote a bug, we'd just have testers left looking at their hands! If we were to lose our jobs over mistakes, we'd all be unemployed! I'm not sure exactly what Schindler's looking for here, but I can tell you I don't like the direction she's going.

Schindler's post is mindless crowd-mongering, aimed at stirring up a crowd (and keeping her name in print). It's also rife with wrong conclusions and, if her assertions were taken to heart, the result would be drastically unrealistic.

But let's ask the question: should Microsoft, and the engineering community within and without, be worried about this defect? Yes! What are the real take-aways from this issue? Read on...

First: Update the Test Repository

The first take-away here is that this is a bug with high potential repeatability. While this is the first known instance of this defect, it does seem possible that it could happen again. So lesson learned - update the test repository. Next time you test leap year handling, be sure to cover not just Feb 28, 29, March 1, 2 but also December 31.

More than updating the repository with this specific test case, testers need to extrapolate a series of test cases. There's something happening here. Schindler might, in fact be right that it's a math bug. It might also just be a buffer overrun (365 days were used up by Dec 30) or something else. The point is, there's a class of bug to be researched and investigated. It's around date management, date accounting, infrequent and semi-regular change and similar topics. And you can bet that both the developer and the feature tester on this component is working on this!

Second: Component Testing

Bill Gates' vision for Trustworthy Computing has always included the concept of distributed programming, where the application model is heavily object-oriented and componentized. The mistake many testers and organizations make is they only want to perform system or functional testing on the completed product. One of the key lessons of Agile and XP development methodologies is that you break things up into pieces and test them. According to the press release, this bug is at the component level and that's where it should be caught.

Organizations need to reconsider their risk-based approach, and ask if component-level testing (unit testing, functional testing: different names for the same activity) would be more effective. Component-level testing, combined with rapid feedback (ie, daily builds with testing involved) will prevent defects from creeping in at the component level, and might have caught an issue like this one.

Third: Take Time to Think

This third point is a personal mission for me. In the goal to minimize investment, IT organizations are constantly squeezing test. Development can overrun dates willy-nilly, but it's always a requirement for the test org to 'make it up. Hey, it's business and we need to hit deadlines. But applying the same effort at the development level (in this case, the effort of increasing discipline to stay on schedule) as the test team is expected to apply (in this case, the effort of catching up when dev has blown the unrealistic schedule) could prevent the need for herculean test effort, and would yield more time for the test organization to think about and improve upon strategy.

Why does this matter? Finding defects like this requires creativity. It requires looking beyond functional test cases and thinking about 'what could go wrong'. Schindler's making the same mistake many people make - overlooking the fact that testers have to catch all of the bugs, whereas the user only needs to experience one. Yes, we sign up for this when we enter a career in quality, but that doesn't change the fact that there's a disparity in expectations. A highly successful, high-quality release cannot be achieved if you force-march your test team and cut corners.

Most IT organizations get away with this approach because their customers are internal and, therefore, are more forgiving. But this kind of bug is very similar to security defects. They're difficult to find, require a lot of creativity, and take time to root out.

So, a great big raspberry to Esther Schindler for a poorly-written article on the Zune incident. True, customers were served poorly. More importantly, the engineering community needs to take lessons here. Her lessons? I question the logic used in drawing her conclusions and I really feel her posting was a populist knee-jerk lacking any substantive value for the engineering community.

At the same time, there is plenty to be learned, and learn we better! Let's build off this new class of defect, let's approach our testing from a more component perspective, and let's think about how much time testing is given to being creative.

Full disclosure: yes, I work at Microsoft as a Senior SDET Lead. But I'm critical of my employer where it makes sense. If I were the manager of the Zune test org (and I know several people in the org), I'd be having the same conversation with them that I wrote in this blog posting.

How Can I Further My Testing Career?

2008-11-26T17:32:00.000-08:00

I recently answered the following question, as part of my "Ask the Expert" role on searchsoftwarequality.com: "I am working as a black box tester on web applications. I want to improve my technical skills and am wondering what type of skills I should learn to help me further my career."

When it comes to testing, as a manager I am concerned about two things: effectiveness and efficiency. Effectiveness is about how well we measure the quality of the system under test (meaning what level of confidence we can gain about its adherence to requirements) and how many defects we discover, fix, and regress. Efficiency is about how many resources it takes to accomplish that task. So when a tester asks me how they can improve their skills, I focus on learning and development which includes these areas.

First, let’s look at effectiveness. Anything that helps you test more deeply, cover more ground, or find more bugs is going to be beneficial. In web testing, I find having a strong knowledge of HTML and CSS, HTTP and HTTP web servers, java script, and now RIA technologies like Ajax, Silverlight, and Flash are super-helpful. Knowing HTML and CSS are self-evident. You want to be able to quickly find errors in layout and styling, freeing the developers from debugging this on their own. Understanding HTTP and web servers is key for lower-level communications issues and for troubleshooting communication problems. Finally, scripting languages are the basis for nearly every commercial web site. Script errors abound (especially because Javascript is a run-time language and really can’t be stepped through easily), and testers need to be able to catch those errors with code review and other methods.

Efficiency to me is mostly about automation—it’s about being in more than one place at the same time. By writing good automation (catches true failures, does not report false passes, runs in many environments and lasts the product cycle), you accomplish several things. First of all, you continue to validate product quality even if you, personally, are not executing the tests. Secondly, you generally execute test cases more quickly—this encourages developers to run these cases even before checking in. You speed up the turn-around time (known as the feedback loop) on the code-defect discovery-defect remediation cycle. Finally, you enable yourself to test on many different configurations at the same time. Common automation skills are familiarity with xUnit test harnesses, ability to use Selenium and Selenium RC or Watij/Watin, and good skills in development in languages such as java and .NET.

You’re probably thinking that I failed to bring up skills in tools such as Rational Robot, Loadrunner, Winrunner and such. These are very important niche skills that a Web application tester can use to differentiate themselves. The really didn’t fit well into my effective/efficient paradigm though! Understanding how and when to apply tools in performance, stress, and security testing realms is a great way to improve your career. The logical step in a tester’s career is to branch out and select specific tools to excel in. If you want to stay in your current company, pick the tools most commonly used for these automated testing categories. If you are looking to move to another company or into another industry niche, understand the tools in common use and develop skills in them. Many commercial tools have limited trial versions available. You may not be able to run a full load test with these tools, but you can definitely learn how to use them. But keep in mind: more important than just learning how to use the tools, understanding what’s happening in your system when you run these skills is even more important.

In 2007, I wrote a series of articles on my blog titled “How to Become a Better Tester”. The first entry can be found here: http://thoughtsonqa.blogspot.com/2007/12/how-can-i-become-better-tester.html This blog series will give you more detailed steps you can take in developing your career.

All the best!

John O.

Lessons for Teamwork in Software Quality

2008-10-22T21:07:00.001-07:00

In my role as a volunteer youth leader in my Church, I have the opportunity to help put together a monthly activity. My group was responsible for the activity this month, and we chose to offer five team building exercises (all straight from http://www.wilderdom.com/games/InitiativeGames.html). We did four exercises in a rotation, and then all met together to perform the fifth. The exercises were:

Minefield: each member of a group paired up. We set up a maze using chairs and tables, and had our pairs blindfold one person. The second person in the pair had to 'lead' their partner through the maze. The challenge in this game is that all partnerships are talking together, at the same time. The purpose of the exercise is to emphasize the need for communication as well as the ability to pick out the voice you're listening to.
Toxic Waste: this is a group exercise. Participants encountered a bucket in the middle of a 10' circle and they're told the circle is full of toxins. They need to move the bucket out of the circle, and their resources are a bungee cord and ten thin ropes. The challenge is ingenuity, teamwork, and (again) communication.
All Aboard! In this exercise, the team all stands on a tarp. The challenge is to fold up the tarp to be as small as possible, fitting the entire team on it, WHILE the team stands on it. Teamwork, spatial relations, etc.
Helium Stick: I conducted this one. The team lines up in two parallel rows, sticks their hands out with their index fingers extended. A small stick (I used a 1/4" wood dowel) is placed on their extended fingertips and they are challenged to lower the dowel. In fact, it goes up.
Egg toss: each team is given 25 straws, 20 cotton balls, a 5' piece of tape and an egg (unboiled). The mission is to build a crate/ship/container so they can drop their egg and it doesn't break.

What is interesting to me is the way teams interacted to get the problem solved. As I said, I conducted the helium stick challenge. As teams started, they were astonished that their stick ROSE instead of sunk. In a flash, right on the heels of that recognition, came frustration. We were split up into groups of boys and girls--oddly enough, while both groups expressed frustration, only the boys started yelling at each other. I kid you not, they were really ripping on one another. They quickly moved into the blame game. Each group (all four) needed to be stopped multiple times.

After frustration/blame, the groups moved into 'try harder' mode. I finally would stop each group and point out that trying harder wasn't working - maybe there was a better way. At that point, it was again quite funny - no single group picked an organized way to brainstorm. It was 'herd brainstorming' and they all just started shouting ideas.

Eventually someone 'won' through all the shouting. In one group, it was an adult leader, who started back into try harder. In a group of 12-year old boys, one of the boys got everyone's attention and he literally talked the stick to the ground. You need to understand the magnitude of that feat - you've got eight people all pushing up on a stick (pushing up because they have to keep their fingers on the stick at all times, and by doing so they force the stick up). Somehow this boy talked everyone, step by step, through the process of lowering the stick.

The girls were pretty creative--they caught on quickly that they needed to coordinate, and touch was the best way to do so. One group of older girls interlocked pinky fingers and lowered together. Another group split into two smaller groups on each end, and just put their hands touching, next to one another.

For me the takeaway was:

If something isn't working, the first response is to try harder. But if it doesn't work, it's not going to work any better if you try harder.
When something isn't working, teams gravitate toward frustration and even blame. It's so destructive! No one was intentionally pushing the stick up, but they kept yelling at each other about it.
Someone has to step up and coordinate the discussion. Brainstorm on ideas and then try one of them -- any one of them. There was no single right idea and the teams that just tried something generally succeeded, as long as it was something other than trying harder.

How can this apply to software development, software testing and software quality? Well, software is built by teams - even the smallest unit of engineering is a feature team made of two or more people. Communication is important, and it's critical that 1) no one resorts to blame and 2) each person is allowed to share their point of view. "Writing unit tests just so we can hit a goal of coverage seems to be distracting us from the real goal--our output is actually lower quality right now than it used to be" needs to be answered with "Why do you think that way?" and not "Uh huh - not true. Besides, you're not helping at all with all these bugs you're bothering me with!"

Good communication includes illustrating the current status, as in "Hey wait, everyone seems to be pushing up" and then a group discussion of what's causing that: "We're pushing up because we all want to keep our finger on the stick." Only then can the group move on and start thinking about the solution. So in an engineering organization, that might be "why are we getting a flood of bugs? Wait, testers are off doing something else (building a battleship test automation system) rather than focusing on testing daily builds". Once the situation is recognized, only then can the team react with an appropriate response.

As quality assurance/software testing teams work with and communicate with development counterparts, quality becomes a natural by-product. As teams discuss what practices are producing current results, they can move forward and improve how they approach the challenge of writing quality software. A little communication can move our engineering teams from try harder to work smarter, and output increases in both quantity and quality.

Technorati Tags: software quality,software testing,quality assurance,qa

What's the Best Software Testing/QA Tool?

2008-10-09T21:35:00.001-07:00

Frequently people will post a question like "What's the best tool?". It drives me nuts sometimes! Software testing and quality assurance are definitely challenging professions, but it's not fair to depend on others to solve your challenges for you! I've answered questions like this in the agile-testing@yahoo.com group, and on the MSDN forum, and I thought it was time I brought my answers together into one blog posting. That way, I can refer people back to the posting.

MSDN Answer

Lifted from my post on MSDN's software testing forum:

An American car manufacturer once tried to make a car for all things - it was a 5-6 passenger car in three different models, each with four wheel drive AND good fuel economy. It ended up mediocre at all things. You need to beware; resist the urge to pick one monolithic tool solution for such a wide variety of testing needs. The vendors (well, NOT Microsoft, of course ) would love to have you believe their tool can do it all. And in some aspects, their software testing tools probably can. However, will that one tool solution be effective and efficient in everything you do? No.

I'm not advocating a pantheon of tools, but you need to think like an engineer more than a manager or a customer. You find the right tool for the job at hand. Over time, you'll settle down to a group of 3-4 tools and you'll stick to them.

I've never used <some tool the asker referenced> - someone else will need to comment on that product's ability to do everything you're looking for. My take? It'd be too good to be true if it actually could. And I tell my three sons all the time 'If it's too good to be true, it's probably not true'.

Be wary.

Here's how you [should] approach this as an engineer:

List out the applications you'll be testing 6-12 months from now
Think about the test scenarios, especially those you'll want to automate
Ask yourself how many releases of that application/project your company will perform--will quality assurance be a repeated process, or are you testing this software just once?
Based on that, how much automation is 'worth the investment'? Microsoft Office 2007 *might* still be running automation I write in 1997--probably not, but I know 2003 runs it AND that automation is still being run, every time there's a patch or SP released.
Now that you know how much investment to make, look for best of breed solutions for each project. Don't focus on all-in-one solutions; just look for the best tool for the job. Use demos, read reviews, ask questions. Don't ask "Is this the right tool?" but rather "Which tool do you recommend" or "I'm considering this tool for this job - does anyone have experience using this tool to do this?"
Once you have a short list of tools, look and see if there is commonality/overlap. You'll see patterns. It's possible Rational or another all-in-one tool will appear in the list; it's equally possible none of those tools will appear.
If there's good overlap, ask yourself what you think the pain will be if you force a tool into a job it wasn't designed for. If you can live with the pain, go for it... If not, keep looking or open yourself up to a larger set of tools.

Hope that helps (in spite of not answering your question),

John O.

Commentary

OK - I searched through a number of documents and can't find my other posting on this subject, so I'll just write it once more.

To all testers: look before you ask. Really! If you are wondering what tool you can use for a given test type, use Live Search and look for information! Performing a minimal amount of research shows respect to the audience you're turning to for help, and can actually prevent a question now and again. QA is all about searching for product defects; apply that searching capability to answering your questions.
You're more likely to get help on a specific question rather than a broad question. For instance, asking "What's the right tool" won't get you much. However "I've evaluated Selenium, Selenium RC, and Watij--given my situation (describe it) what tool do you recommend?"
Talk about the problem you're trying to solve. So "We realized we'll be running tests over and over and over again, but our UI changes frequently. What strategies can we take...?" is a question that raises the problem and let's people know what help you're looking for.
Asking "what's the best tool?" is like asking "What's the best car?". If you live in Germany and drive the autobahn frequently, the best car for you is far different from the best car for someone who lives in Bombay, battles thick traffic, and drives Indian roads (notoriously rough). Be specific, give details, and look for recommendations. Software testing tools are the same way - a screaming "BMW" test tool will fall apart on a Bombay road. A right-hand drive Jaguar would be totally out of place on an American highway. A performance test tool isn't the right way to automate repeated quality assurance tests. The right tool is dependent on the project, the skill sets on the QA team, the timeline, and several other factors.
Solve your own problems. The Internet is an incredible tool and offers us all a ton of opportunity to not have to reinvent the wheel. But don't ask other people to do your work for you! Ask for advice. If you want someone to solve your problem, ask one of us to consult (for pay) and bring us in. We'll be glad to get the job done for you!
Give back: as you grow and learn, give back... Don't post a question, get your answer, and disappear. Remain an active participant in the community and 'pay it forward'.

Summary? Be specific, research before you ask, solve problems, and give back. That's how to get answers online--in a sustainable fashion.

Technorati Tags: quality assurance,QA,Software Testing

What Makes a Good Automation System (automated QA/Quality Assurance/Software Testing)

2008-10-03T07:45:00.001-07:00

So in my new job, one of my first tasks is to put together an automation system--by this I mean a harness and a framework. The process has had me thinking (and talking) a lot about what makes a good system in general.

The automation harness is the system used to schedule, distribute and run tests and to record results. In the open source world, some tools used here include NUnit, JUnit, and TestNG (my personal favorite). These tools all work in a one-off situation - they are all run locally out of the dev environment or via the command line. In software testing at Microsoft, though, a one-by-one approach to automation is useful for 1) developer unit testing, 2) tester unit testing/test creation, and 3) failure investigation. However for the 24/7 test environment we're building, this isn't sufficient. We need a centralized scheduling tool that allows us to push tests out to multiple clients (simulate load and de-serialize test runs). So we're working internally to Microsoft, evaluating the existing automation harnesses available and trying to find the one that works best.

A big factor for me in this selection is finding a harness which is configurable. At Microsoft, we are pushing the envelope in testing in a variety of ways: code coverage analysis, failure analysis, and several similar activities which allow us streamline our testing, reduce test overhead and automate many testing tasks. This means our framework MUST be extensible - we have to be able to plug in new testing activities.

A second element of the automation system is the framework. This is the abstraction layer which separates our automated tests from the application under test. This abstraction layer is critical to good automation. If, for instance, you are automating a web application, you will probably experience a lot of churn in the application layout. You do not want your automated tests hard-coded looking for certain controls in certain locations (ie, in the DHTML structure)--by abstracting this logic, your test can call myPage.LoginButton.Click(), and your abstraction layer can 'translate' this into clicking a button located in a specific div. In some organizations, this framework is purchased. At the LDS Church, we leveraged both Selenium RC and Watij to build this framework, developing most of it ourselves internally (kudos to Brandon Nicholls and Jeremy Stowell for the work they did in this capacity).

The challenge felt by most test organizations is two-fold: 1) finding the engineering talent to build these systems and 2) making the investment in innovation. Ironically, the very thing which can free up resources for other tasks (automated testing) is the thing most managers don't want to spend time on! This makes sense, sort of... managers don't like to invest in activities which (in their mind) don't contribute directly to the bottom line. In all but the smallest of projects, this makes no sense--test automation isn't a sellable product, but if automated tests can free up a day or two of test time, that's a day or two spent doing other activities! Each and every time automation is being run.

Recruiting top talent is also a challenge. In both IT organizations where I worked, there was a culture among developers that testers weren't engineers--they were click testers. Testers couldn't give input on 'extremely technical concepts' like architecture, potential bug causes, or the likes--they're there to pick up code as developers release it, and then to find bugs. It's no wonder that it's so challenging to hire engineers into testing - when they're treated like that, they're either going to leave or move to development!

So the keys to a great automation system are: 1) a solid, extensible and flexible harness, 2) a robust framework, generally customized to your test activities, 3) management commitment to invest in innovation and automation and 4) top engineering talent and the culture to reward them for their contribution.

Am I missing anything?

Technorati Tags: quality assurance,qa,testing,software testing,automation,test automation

What A Feeling!

2008-10-01T07:00:00.001-07:00

Technorati Tags: Software testing,quality assurance,qa,engineering

So I've done a relatively good job of being calm and collected in my first two days at Microsoft. Your first day starts in a long line filling in I-9 forms. Then you enter address and other contact info info the Microsoft system via an internal web form. Finally, you sit for another 7 hours in a room while they teach you about benefits and the MS culture. At the start of that first day, you're officially a Microsoft employee, but you don't get your card key or anything.

Day two starts in the Big Room again, with a discussion about corporate ethics, legal issues, and a discussion with a couple of recent Microsoft hires. Finally you get your card key and are sent to find your manager. Oh and by the way - the room has anywhere from 100 to 150 people in it. Yup - Microsoft starts that many people, each and every week (well, maybe not the week of Christmas or New Year).

So at about noon I launched off on my own. Unlike most new hires, having worked here for 11 years, I know where the buildings are, where parking is, etc. So I zipped straight to my temporary building whenever I'm here in Redmond. I swiped my badge and got access to the building. A little smile crept up on my face.

About an hour later, after picking up my laptop and getting it set up, I went for lunch with another new-hire for our Utah group. As I swiped my card and stepped into the cafeteria, I had a completely involuntary reaction: I jumped, throw both hands high in the air, and shouting "Yes!! I'm back at Microsoft!" Later during lunch, Tim told me how cool it was to spend time with someone who is so excited about working at the company. I have a bounce in my step which has been missing for many, many years.

I can't describe it. I was in software testing/quality assurance at Microsoft for 11 years. There were great days and there were really challenging days. I left two years ago to lead QA "for the world's largest retail IT project" at Circuit City. What an experience that project was. Quality assurance to team leads (mostly IBM project managers) meant proving happy path and avoiding negative testing. It was a cultural shock, to say the least. Testing software at the LDS Church was somewhat better. The people, for the most part, were great (surprisingly, there were exceptions - people who behaved less Christ-like than even at Microsoft!). The QA team suffered from a total lack of respect from development, however. And I found in that IT organization that everyone has a special little niche. There are enterprise architects, application architects, security 'specialists' (people who know about security policy, but don't know much about penetration testing), developers, and quality assurance engineers. If you dared to stray outside of your niche, well, that meant you were stepping on someone else's toes.

Back here at Microsoft means I have an equal seat at the table as an engineer. It means I'll have to work with other engineers to tackle really, really challenging problems. First challenge: building an automation harness using existing technologies at Microsoft, then building our own framework (abstraction layer) within which our tests run. Additionally, we need to take the mandate to "build virtualization management technologies" and turn that into a release software application: ideation, product planning, product specification, development, and release testing.

We also have the challenge of hiring incredible C++ developers and testers (engineers) in the Salt Lake Valley. Finding developers is pretty easy, but finding developers who respect QA and understand engineering excellence? A challenge. Finding a software testing professional who has the guts to take an equal seat at the table? A challenge!

But that's what I love about being back. I feel like, after two long years, I can finally do my best work and reach my full potential. I can bring all of my 14 years of engineering experience to bear on a software challenge. If I have an idea, I can run with it. I can provide input to the user scenarios, to application architecture, and to how we push quality upstream. I can prevent software defects, rather than find them!

I know everything won't be perfect. I left Microsoft for reasons, and those reasons haven't all changed (although judging by much of what I heard in New Employee Orientation, the last two years have been a time of growth and improvement for the company). There will still be bad days, there'll still be the struggle for a good work/life balance (now called 'blend'). But I have incredible health benefits for my family, stocks and bonuses again, and I have the chance to be challenged every day again. And I'll be working with super-smart people every single day. THAT is a cool thing!

A Softie Again

2008-09-22T21:23:00.001-07:00

I'm thrilled to announce that I'm about to become a Microsoftie, again! I have been given the opportunity to return to Microsoft as a senior test lead, working in the Management Division. No, that's NOT the group of execs who run the company! That's the group producing OS and platform management tools like Operations Manager (a product I've worked on in the past), SMS Server and so forth.

About two months ago, Microsoft announced the formation of a development center here in my home of Salt Lake City. I quickly applied for roles in the test organization. After two years of working project IT, I am ready to get back to product software! Some people might snicker, but the commitment to quality at a product software company, especially Microsoft, is so much higher than in project IT. I found I was spitting in the wind quite often in my previous roles--I was either fighting losing battles or I was fighting the wrong battles (ie, pushing for quality in organizations or situations where leadership didn't share the same commitment, or pushing for higher quality than some projects required).

Additionally, I found that my last few organizations did NOT look at testing engineers as equal citizens. Testing was looked at primary as QC (quality control) or possibly QA (quality assurance). Rarely were we invited to planning or design meetings, never were we looked upon as people who could help PREVENT defects. We were often seen as a roadblock to release. Don't even ask about my two-hour argument with IBM about the value of negative testing! Engineers who could help architect solutions? No way!! I often felt my fourteen years of engineering experience were swept under the table because I had "QA" in front of my "Engineer" title, rather than "Java" or "Development".

So I'm very excited to be returning to an organization and a company where I get a 'full seat' at the table, and an opportunity to contribute to my full ability, not to the perceived level of my title. IE, I'm an engineer again!

Doesn't change much about my point of view or my approach to testing. I'm still hyper-focused on 1) bug prevention, 2) quality engineering, and 3) driving for greater effectiveness and efficiency. And sometimes that will put me at odds with mainstream MS thinking (recall my conversation about moving to all SDETs and automating all tests, and the resulting "user experience" bugs that slip through the cracks in products like Vista).

For me the best thing is, I get to stay in beautiful Salt Lake City, and yet work for one of the greatest employers in the world. And NOW I can honestly say I've tried a few others...

So OK - let's hear it. How bad is Microsoft quality, really? If you could talk to a test leader, what would you tell them? What would you ask them?

Technorati Tags: software testing,quality assurance,QA,Microsoft

Software Quality Assurance: SDL (Secure Development Lifecycle

2008-07-25T22:25:00.001-07:00

At work, I've been helping one of my teams implement portions of Microsoft's Secure Development Lifecycle (SDL). SDL is more than just plinking around attempting some penetration testing--it's a committed approach to secure software design. Here are some of the takeaways from our work:

The first point I made with my team is that security features DO NOT EQUAL secure features. Having SSL encrypted communications does not make a web application secure! It just means you have an encrypted communications channel. Secure software isn't secure because of features such as Acegi security, RSA encryption or anything like it. Secure software is produced when developers think secure from the start. Secure software comes when code is written safely, when developers write solid, secure code, and when testers are helping with the planning and testing of the software with a secure focus.
Next point we emphasized: threat modeling. As we wrapped up a two-hour threat modeling session, one of the developers commented "Why didn't we do this months ago, before our code was written!". Good point!! It's never too early to threat model. Analyze your product, paying special attention to where data crosses boundaries: user to Internet, Internet to server, server to database, etc. Model threats, wild and crazy or down-to-earth. Our threat modeling has resulted in 9 potential threats so far, and we expect several more as we continue.
Security comes in layers. Back when I lived in India, I toured the Delhi fort. This fort was built by professionals. It has a deep moat around it. Tall, thick walls surround an inner wall, and inside that inner wall lies the fort. That's how our code should be! OK - so you're authenticated via Acegi and LDAP. You're encrypted with SSL. That's great - but what if someone logs in with a valid account, then tries to hijack another session? Your layered security will catch hacks like this--authorize anytime someone tries to access sensitive data. Even if you've already authenticated and authorized, do it again! Layers bring security.
Reduce your footprint: in Agra, where the Taj Majal is, there's another fort (these Moguls were building forts everywhere!). This fort is on the edge of town, in the hills. Compared to the Taj, the fort is tiny. This is referred to as attack surface reduction. Only allow public access to a few of your resources. If you have features which suffer from weak security, disable them by default or remove them completely. Give hackers as little space as you can.
Train your engineers (dev and test). There's common training needed by both (elements of secure design, running a threat model, etc.) and there are discipline-specific trainings such as penetration testing or the application of specific technologies. The SDL is called a lifecycle because it's a continuous process. Lather, rinse, repeat and all that.

Our training has produced benefits. For starters, developers have a new security-focused mind set. We've found a few security bugs already, and our threat model has exposed some potential issues. This is great progress, and it comes from just one day of work. Imagine what we'll be like in a few months after a day or two of training and a complete milestone with security in mind!

It's never too early or too late to take a step back and start thinking security. I designed our course based on my experience at Microsoft, which was neatly documented in Michael Howard's new book "SDL: Secure Development Lifecycle". I cannot recommend reading this book enough!

Got a security question? Post it here...

Technorati Tags: software testing,qa,quality assurance,security

Software Testing Patterns: Sessions

2008-07-15T23:25:00.001-07:00

One of the most common things to have to test is sessions, cookies, and login/logout. This is the start of a pattern for that kind of testing.

First, some background:

OWASP has a great cookie/session state document: http://www.owasp.org/index.php/Testing_for_Cookie_and_Session_Token_Manipulation

Wikipedia HTTP cookie: http://en.wikipedia.org/wiki/HTTP_cookie An excellent treatise on the origin of cookies, with a synopsis of cookie attacks.

How to view cookies in IE 7 http://kb.iu.edu/data/ajfh.html

Test Patterns:

Session hijacking: create several cookies for your site (varying login sessions). Tests: 1) can you substitute the session id from one session and use it in another session. 2) what happens with invalid session IDs? 3) Prevention: specify that cookies must be https-based (specify the secure flag when creating the cookie.

Cookie poisoning: change values in a cookie - for instance, if a cookie stores the check-out value in a shopping cart, the user could change that value before checking out.

Back button: do something that adds/modifies a cookie, then click 'back'. The state could then be out of sync (being on a page which expects you to NOT have a value in your cookie which you actually do have).

Expiration: cookies w/o expirations are considered 'not persistent' and should be destroyed at the end of the session. Persistent cookies represent a security risk due to how long they are persistent; make sure the expiration of a cookie is reasonable (30 min?) based on typical user scenarios.

Log in, thereby picking up a sessionID. Make note of the sessionid. Close the browser, and then log in as someone else. Finally, update the cookie to be the id you noted previously, and hit refresh. Does the session change?

Software Testing Patterns

2008-07-15T23:25:00.000-07:00

I'm convinced there are a series of testing patterns we could use. I haven't found any on the Internet, which is surprising, but I figure I'll start documenting them here. Look for blogs with "Test Pattern" in the title.

Don't be shy about contributing, please!

First of all, some links:

Open Web Application Security Project (OWASP) http://www.owasp.org/index.php/Main_Page Very active, even developing automated test solutions and such. Tons of list serves and such.
Web Application Security Consortium (WASC): http://www.webappsec.org/ Not all that active, it appears, but there's some interesting info.

Quality Assurance or Testing Seminars In India

2008-07-15T21:57:00.001-07:00

I've often wondered if there's a market for QA/quality assurance/testing seminars in India. I've thought of setting up seminars for how to test in cities like Mumbai, Bangalore, Delhi, Chennai. One day seminars where people pay a flat fee to get in and we spend the day really digging into testing. Some presentations, some 'labs' and a lot of Q/A.

If you think there's value, post a comment. If you'd be interested in partnering, hit me in an e-mail joatgm at gmail dot com.

Should Freshers be Testing Software?

2008-07-15T21:33:00.001-07:00

I have seen an alarming trend in India and worldwide. "Test engineers sought - freshers" (for those not familiar with the Indian IT scene, a 'fresher' is someone fresh out of college - 0 to 1 year experience, maybe even a little more).

Seems like everyone with an open tester position wants to throw a fresher at it. This is a huge mistake! What does a fresher bring to the table?

Enthusiasm
Lots of book learning; in India, that means rote memorization
Some programming experience.

I once helped a large, failing consumer electronics company move work offshore to save money (offshoring to save money is a BAAAADDDD idea--ask me why or wait for a blog on it). The development 'partner' on the project brought in a ton of freshers and some somewhat seasoned testers. What did we get? We had test engineers who didn't know what a defect report was, who'd never used a defect tracking tool, who had absolutely no domain knowledge. All they had to contribute was 1) a warm body and 2) some experience in Java. This is a recipe for disaster.

Is there room for freshers in an IT organization? Absolutely! Microsoft focuses most of its recruiting in the college space, but that's because the ratio of new hire to full timer (at least in the US) is probably 10:1. They have people all around them who can mentor and grow them, and they are NOT encouraged/forced to do mundane and boring tasks. Well, not all the time... As a new hire at Microsoft, I was the release lead for German Mac Office 98, with 18 months experience in the company! But that was after 18 months of incredible mentoring and learning.

For you people hiring for testers, don't think "Well, no one wants to be a tester - let's find someone who wants a foot in the door and get them in here." You're doing yourself a disservice, you're doing your customer/project a disservice, and you're really not helping the fresher either. Look for some passionate QA/testing professionals. Bring them in and let them establish a world-class test organization. Then hire freshers, train them up, and keep growing your organization.

OK - off my soapbox for now...

Thanks to the Communities

2008-06-12T12:26:00.001-07:00

I belong to several communities - Agile Testing group, Watij group, MSDN Software Testing Forum, etc. I just wanted to blog a quick thanks to all the active members of these communities. Just in the past 6 days, I have:

Gotten help getting Watij up and running and implemented
Gotten sample code to write a tool which dismisses error dialogs that are popping up in our automation
Gotten answers to several other questions

Engineering is tricky. Civil engineering is a moving field, but it moves a lot slower - advances in concrete, paving techniques, etc are probably all documented somewhere in a journal each month. Changes in technology? Rapid pace! I don't think any of us could keep up with all the platform, open source, commercial tools, and techniques which change over time. But by having these communities, we are able to give each other a leg up.

Take my dialog dismissal app... We're building automation in Selenium RC (Java). There's an issue in Selenium's Chrome browser that you cannot permanently accept a cert. Every our automation switches into SSL, we get the warning. Until today we have been unable to run all 350 tests in one setting because none of us wants to dismiss that dialog 1000 times. With some sample code from Michael Johnson via the MSDN Software Testing Forum, I put together a quick C# app that watches for and dismisses this dialog. Problem solved, and for the first time ever, we can run our automation end-to-end.

Thanks folks, for all the effort! We make each others' jobs easier this way. Kudos to everyone who puts in time and effort on open source tools like Watij, Selenium, TestNG and the likes, too.

John O.

Advice to Developers from a Test Manager

2008-05-30T06:31:00.001-07:00

Technorati Tags: testing,qa,quality assurance,engineering

A good friend pointed out that a blog on advice to developers from an experienced test manager would be helpful. With 13 years of tech experience now, I have a good idea of what works and what doesn't.

Most Successful Projects

I'm basing my advice on my experiences in the most successful projects I have worked on. These projects have been released on time, have had the highest possible quality, have thrilled customers, and have done so with minimal pain to the entire engineering team. These projects themselves are:

Microsoft Server ActiveSync: synchronization layer between Microsoft's mobile devices (Pocket PC and Smartphone) and Exchange Server.
Microsoft's Learning Essentials for Microsoft Office, a free enhancement to Office to help tune it more for the needs of students and teachers.
The LDS Church's Its About Love adoption site (unreleased, sorry!)

So let's jump in, shall we?

Collaboration

Successful projects all share the element of collaboration between development, test, and program management/product design. There's a feedback loop in this relationship which results in higher quality. Dev can provide input into design tweaks which might result in significantly lower development effort. Test can provide feedback which enhances testability and reliability.

During the engineering process, this collaboration continues both in refinement of the feature list and of features. That close relationship between dev and test also helps improve quality through quick defect remediation as well as mutual support in defect detection. When dev and test work together (as opposed to dev working in a vacuum and throwing something over the wall), invariably the result is fewer bugs, more bugs fixed 'right', and a significantly shorter bug turnaround.

I want to stress this point. It is so very critical for bugs to be detected and fixed early on. The longer a bug remains in the project, the more code there is which is written around that bug. It becomes like a sliver under the skin - the code literally festers, becomes infected, and removal/healing are hindered. This quick turnaround simply can't be called out enough. It's definitely a must-have.

An additional aspect of collaboration is seen when dev and test work together to find, investigate, and remediate defects. Frequently this takes the form of "Hey Tester - I checked in some code today, it's all passing unit tests but I'm a bit concerned about it." Just a heads up generally points a tester down a different path than they may have planned to take.

Paired investigation can be so critical. In It's About Love, we have found a series of strange performance issues. I've got the time, experience, and tools to find the bug whereas my developer has the experience to tweak his code to find potential solutions. It's a symbiotic relationship. Without this paired work, he'd be blindly trying to fix something, throw it over the fence, and get it back the next day as a failure.

I list collaboration first because it is the key. Everything else you do to improve your product and code will stem off of collaboration, so you had better learn this skill if you want to be an effective, efficient developer.

Trust Your Tester

Ah, trust... that word of words, that concept of concept. I climb mountains, or at least I did until my kids got older. Now we hike, and someday we'll climb again. It's taught me a lot about trust. When you are scaling an ice field, your life is dependent on your skill and strength. It also lies in the hands of the person at the other end of the rope. You learn to trust your partner explicitly--you have to!

Some developers look down their nose at testers. Maybe it's because the developer feels they have more experience writing code. Maybe its because the developer has a degree in computer science and the tester does not. Maybe its because the developer feels he or she 'won the lottery' and the tester did not. It may even be that the developer feels like they are in a master/servant position. Believe me, these feelings will not help your project, not in the least.

Hear me on a few things:

There's no master/servant relationship. The tester does not exist to beat quality into your rushed, careless code. If this is your attitude, the first time you run into a testing professional, you are in for a rough time. There's nothing sadder, in my opinion, than a dev who rushes through their code, cannot be bothered by incoming defects, and then throws the whole mess over the fence to a tester to root out the issues. That's pure laziness, it's pride, and it's a lack of professionalism. If this is your approach to engineering, you seriously need to rethink your value to your organization. You also need to step down off your high horse and start to think about the value in a team approach. You need to learn to take pride in your results - not just that you threw together a bunch of web pages or produced an application thousands will use. Will they enjoy their experience? Is it the best thing you could have possibly built? If it isn't, wouldn't you rather learn to make it that way? Western society is built on the concept of pride in outcome, of making highest-quality products. Don't push together a bunch of crap, throw it over the fence and expect QA to work it out. Or worse, don't get annoyed when you product a lousy piece of software and testing has the audacity to find a bunch of bugs in it! Just because it compiles and passes your unit tests doesn't mean it's worth the cost and effort spent on it!
There's no lottery. You may be performing your dream job, and you may think being a tester would be the worst thing that could happen to you (equivalent to, say, working a cash register at McDonalds). Believe me, there are a lot of 'testers' who would rather be developers. But most testers have chosen this career path and they are as passionate and excited about being a tester as you might be about being a developer. Think about where the US Space Shuttle would be without QA, or the Lexus automobile... QA professionals love their chosen career. I, for one, would rather drive a city bus than be a developer. Don't get me wrong - I love to code, it's fun. But to sit in front of a blank screen day in, day out, doing the same thing over and over and over... Yuck! I don't know how you do it! I wouldn't trade places with you for anything. So you may be happy in your role, but believe me - there's no need to pity a tester, nor to look down your nose at one.
Your degree doesn't mean much. I once interviewed a candidate for a development position. He had a a Bachelors in CS from a prestigious nationally-recognized school, and had won multiple national competitions in programming. Unfortunately for him, he had no clue how to write elegant code. I have probably interviewed a thousand candidates, from PhDs to bachelors in CS. And I've hired maybe 50 or 60 total. Your degree is evidence of a lot of hard work (well, maybe...) and definitely of perseverance, but it meant nothing the day you left school and started working for pay. I have a degree in German Literature and International Relations, but I can roll up my sleeves and dig into code and push quality into a project. It takes so much more than a degree in CS to make a good programmer, and some of the best programmers I've known don't even have CS degrees, or earned them as an afterthought. Sure, you gained some experience when you took your CS classes, and you got some exposure to programming theory. But what counts in software is experience, intelligence, and diligence. And your tester can have all of that and never have received a degree, or could have a PhD in physical therapy. Learn to trust the experience. I had a developer rip a test plan up one side and down the other because it had holes in it--that's great. But her excuse for not wanting to discuss the plan was "I have five years of experience - believe me, this is wrong" Well, that's just great, but the point is, she lost me there. At the time, I had 12 years of experience. I may not have known the ins and outs of the code (I was new to the project) but I know quality and I know we weren't approaching it right.

So when you're thinking about your project and questioning whether you should let that pesky tester into another meeting, don't listen to that voice that says you're better off without her. She's going to bring experience, passion, and perspective you simply don't have.

Stick to the Basics

I've been learning a universal truth in life, and that is that things generally don't happen all at once, but step by step. It truly is the little things that matter - it's the basics that count. You know, there's a reason why so many teams are jazzed about Agile, XP, and scrum. In many instances, these disciplined changes introduce quality and drive projects to a faster, more successful completion.

A lot of developers have glommed on to Agile like batter on a bowl. They love the 'freedom' of no rules, no documentation, no meetings - just pure coding. And who wouldn't? Unfortunately for them and for the projects, they don't understand: there are rules in Agile. Agile is based on XP. There are fundamental rules in XP like you don't move ahead without stabilizing what you've got, you don't cut corners, you work in pairs, you don't write code you don't need. These are basics that, when they are ignored, spell doom for your project.

As a test manager, my job--my very reason for being--is to make sure the project completes with highest quality. I'm going to be a stickler for the basics. I once worked two concurrent projects. They both started at about the same time. They were both green-field projects (white-monitor projects?). They were both staffed with some of the best people in the organization. One project was a short-term, single iteration. The other project was a three-cycle, year-long effort. Both projects held frequent scrums where they went around the table and talked about progress. But that's where the similarities end.

The difference in approach was astounding, in spite of all the similarities. The team on the short project cut corners. When the encountered a roadblock on a given user story, rather than spiking in and working through it, they moved to another story. Soon, 95% of the stories in the project had been started, and none had been completed. No master build was produced; devs were checking in code after self-hosting it. No unit tests were developed ("we're too busy coding to write tests" was the excuse). Critical infrastructure tasks such as URL rewrite or SSL were excluded from the development phase because they were 'operations' tasks. Test was uninvolved because 1) test was understaffed and 2) there were no builds to use.

The other project was difficult to start. Testing was involved from day one, and we insisted on adherence to standards. Ironically, we discovered that, while .NET web services had a standard of strong typing out of the box, Apache's CXF does not. We had to figure that out. We had performance issues. But we dug in deep and worked hard - development put in an incredible showing. Test as well, in spite of being understaffed.

Can you guess the results? The first project released, barely. In the final four weeks of the project, official builds were produced and literally 350 bugs were opened. The code churn was unimaginable. Regressions were high--we were at about 2:1 (for every two bugs regressed, either one was reopened or a new bug was found). The product was released, but came in a month late and still ended up in a severely restricted beta. Developers worked insanely long hours, rather than cutting features, because no one story was less than 75% or 80% completed; there was nothing obvious to cut. The program manager got beat up over the project. I'd call it anything but a success.

The year long project? Cycle 1 of 3 was a solid foundation. Cycle 2 forged ahead - there were some challenges integrating, but two weeks of extended time worked through all that. There are performance issues yet to address, but Cycle 3 is going strong, the project looks great, and the team has really banded together.

Don't ignore the basics. You're not immortal, omnipotent, or above the basic rules of engineering.

Take Pride in Your Work

Related to focusing on the small things is the concept of taking pride in your work. I have to laugh at how many times in one of my web application projects I have entered a bug against the application which needed to be fixed for FireFox or IE. The developer would quick-like code up the fix and throw it back to test. The first thing I would do is regress the bug in the browser it was found. Generally worked like a charm. The second thing I'd do? Test it in the other browser. 5 times out of 10, can you guess what happened? It was broken.

No developer worth their salt should be beyond testing their fixes in multiple browsers. In the open source, Java/Oracle-heavy environments I've worked in for the past two years, I've seen a lot of FireFox fans in engineering organizations. That's cool - I like it too. But only 30% of your clients are using FireFox, generally - that means about 60% of them are using IE 6 or IE 7. If you fix a bug in one browser, you still have to look at it in the other two browsers. It is astounding to me that, after 5, 10 of these bugs being bounced back, developers still can't get it.

If you are going to spend time fixing a bug, take pride in your work - fix it all the way, and make sure it's fixed before checking in your change!

Value Your Tester

OK - so you're XP and you write a lot of unit tests. That's great! Congratulations, you've taken the first step to becoming a great developer. But now it's time to recognize that there are still more important things to do. You need realize that quality doesn't end when you finally pass your unit test--just like, when building a house, you're not finished framing when you cut the two-by-four.

Testers bring a very unique perspective because they can generally think in holistic, systemic terms. As a developer, you're thinking (or should be thinking) in terms of methods and functions, services, etc. A tester is thinking in terms of code and database, interface layers, and user interaction. You tie methods and functions together. A tester makes sure entire systems together.

Want an example of this? Your unit tests stub out data coming in and out of a database, to isolate the code layer. You prove your function works, but you don't prove it works with your database. Testers deal with real-world data, and they make sure it comes from the database.

Need another example? Your job as developers is to achieve something - it's to create something of value which fulfills a requirement (a user story). You can prove you are finished because you can demonstrate the functionality of your user story. You start at a broad point (there is any number of ways to accomplish your programming task), and drive narrower and narrower until you accomplish your work. Testing is just the opposite - testing starts narrow (with your completed user story) and goes broader and broader because there is an infinite number of test approaches. Testing can go forever.

Another way to look at it is this: your work culminates in one completed user story. A tester's job can potentially never end, because there could be an infinite number of bugs in your code (it seems like it sometimes!). It is the old turn-of-the-phrase "You can count how many seeds there are in an apple, but you can't count how many apples there are in a seed".

Testers are trained in this. They are used to this and a good tester is comfortable in this. This mind set is opposite to how you work - testers are different. If you embrace and welcome that difference, you will be more successful. Value that difference and watch what can happen as you collaborate on architecture, on infrastructure, even on code design.

Defect Metrics

2008-04-10T22:34:00.001-07:00

Technorati Tags: quality assurance,qa,testing,software,engineering

I had a friend ask me recently about bug metrics, how they're used, etc. I am breaking a personal rule about editing and just shoving my e-mail comments to Brian into my weblog - I generally take more care. But I noticed it's been almost a month since I've posted. I've been so busy (as my mother-in-law used to say, up to my ears in alligators), but that's no excuse!

These articles are all going to end up in a book sometime soon. I'm currently looking for a publisher, and working on an outline. So I'd love your feedback on this stuff.

Defect Metrics

The generally accepted standard is to apply severity and priority. Typically severity 1, 2, 3 and priority 1, 2, 3. NOTE a high severity or high priority bug is actually a LOW value, so a sev 1 is a really critical bug, a pri 3 is unimportant. Severity generally relates to the impact the bug has on a customer. Sev 1 is generally data loss, blocked functionality, etc., sev 2 is generally major annoyance but with a workaround, and sev 3 is a UI, pixel-pushing bug. Generally. Priority relates to the importance of a bug—for instance, fixing a data loss issue is a pri 1, fixing a pixel-pushing bug is a pri 3.

At the Church, we’re currently chatting about changing that a bit. We have a guy who proposed something very similar to what QA Associates (now part of Tek Systems – uck) proposed in an old white paper. That’s the concept of a matrix. At the Church, we are talking about matrixing severity (the impact of a defect) and frequency/exposure (how likely/often a customer is to encounter the bug). Based on that matrix we get a priority. All P1s have to be fixed immediately because either 1) they block further testing or 2) they at least mean we’ll continue to code on a sandy foundation. It’s a unique approach – the objective is to reduce time in bug triage/scrubbing to about 0, and to improve the overall quality because 1) our bug priorities are more granular, 2) the emotion of which bug to fix has been eliminated, and 3) we hold ourselves more to a bar (P2 and up all have to be fixed, for example). We haven’t presented this to management yet—hoping to wrap it up next week and present shortly thereafter.

What metrics are important? Sheesh, could probably write an entire book on that. I really geek out on metrics, actually – something which surprised the teams I work with. They don’t know what to do with me. Here goes!

Income rate: how many bugs are being opened per day (or per build, but per day is easier to measure). Watch that count increase as development begins and test becomes more familiar with the new features. Toward the end of the project, that number had better drop—in project IT, it tends to drop off quickly, in product software the tail is much more shallow. Some of the drop-off is due to being pushed to other projects, some of it reflects hitting the ‘acceptable’ quality bar (which is always lower in project IT than product software) and some actually reflects that the bugs are pretty well shaken out.
Resolved rate: always good to keep an eye on this – is the resolved (but not closed) rate creeping up? Tells you test is not keeping up. Next, you have to ask yourself why… Is test lazy? Are they so busy ‘working’ that they can’t test? Did they have a huge spike of bugs found the previous week and are they just not able to chew through regression as quickly as dev is fixing?
Bugs opened per day, by severity – so interesting to watch! See how many S1 bugs come in from start to finish, esp compared to S2 or S3. In the early phases, most of your bugs had better be S1’s. This is the core architecture, and your testers had better be focused on rooting out core issues. As the income rate starts to drop, you should see P1s dropping and P2/P3 picking up. That shows you’ve stabilized core components and bugs are either coming in from peripheral components OR they are just niggling fit ‘n finish bugs.
Same for priority (bugs opened per day, by priority)
Pie chart: severity: always interesting to see how many bugs are S1, S2, and S3. If I am in the last week of testing, and we have 70% S1 bug count, I *know* we are not done testing. We recently did a study and found 50% of the bugs in all of the Church’s databases are S1. That tells me 1) we write really lousy code, or 2) our testers mark S1 bugs incorrectly or 3) we stop testing before the S2 and S3 bugs are found. A lot of it is the second issue… The project teams here all set S1 as their bar; S2 bugs rarely get fixed. It’s part of people wanting to ship fast. In order to get a bug fixed, therefore, testers have to artificially inflate the bug’s severity/priority. A healthy project will be around 40%, 40%, 20% or maybe even 30%, 30%, 30% distribution. If not, dig deeper and find out why.
Pie chart: priority: pretty much the same. It’s interesting to see the distribution of bugs and how they shake out in priority.
Bugs opened per area: what’s your buggiest feature set/area in a given project? As a lead/manager you’ll want to add some test focus there. As a manager, you’ll start harping on your dev manager to figure it out.
Bugs opened per developer: this isn’t really fair all the time. Some times developers get saddled with really bad legacy code, or they get very complex interface features. But still, looking at the bugs per developer is interesting.
Bugs opened per tester: hey, you can measure bugs and that’s a good thing! A lot of people are sensitive to comparing testers (see the reasons above), but still – the best testers are the ones who find bugs and influence to the point where they are fixed. After all, we pay testers to find bugs right? Well, actually that’s wrong. We *should* be paying testers to prevent defects by helping development NOT check in defective code. But barring that, finding them is good too.
Pie chart: resolution: take a look at your resolutions (fixed, won’t fix, duplication, by design, etc.) and see how your bugs lay out. I was SHOCKED to see we have had a 90% fix rate in a current project, but a lot of that is because we’re still finding P1s. Generally 75% to 80% fix rate is normal—you don’t want to fix them all! If you have time to fix every bug, your schedule is way over-estimated. Plus each fix represents a potential of one, two, or even three MORE bugs introduced. So keeping the fixes down is a good thing.

I'll get some charts on these metrics and post them up sooner or later. Meanwhile keep the comments coming.

Sharing Answers to Interview Questions

2008-02-11T18:10:00.001-08:00

Technorati Tags: testing,quality assurance,qa,interviewing,recruiting

The other day I was looking up some test-related topics on the Internet, and I came across an entire thread dedicated to interview questions and their answers. Readers were enthusiastically thanking the people who posted the questions, making note of answers and thinking this was going to further their career.

Now, I'm all about being prepared. When I prep for an interview, I generally ask a friend to mock-interview me, to get me into the right mindset. While at Microsoft, I was a frequent interviewer but still found it a challenge when the tables were turned. Mock interviews hgelp remind me of the types of technical questions I will probably face, the need to speak slowly and clearly, and above all, how to stay cool under pressure.

There is a difference, however, between mock interviewing and trying to memorize an answer to an interview question. I've seen this happen during campus recruiting (at some very prestigious schools, mind you, where you think the students would either be beyond that or they'd be smart enough to know it's not going to work). Memorizing the answers to interview questions is dishonest, it's a disservice to the interviewee, and it's a disservice to the company.

Let's tackle the honesty part, first. In the US, there is currently an epedemic of dishonesty in our schools. Kids regularly cheat -- they've done it throughout the ages. But now the kids who are cheating actually feel no compuction about it! They find nothing morally wrong with sharing homework, or even cheating on a test!

In the workplace, the same standards of honesty seem to apply. Everyone has had the co-worker who takes credit for work he or she didn't do. The misfortunate have worked for the boss who misrepresented his work or flat-out lies about a pending promotion or the likes.

Because it's mainstream doesn't make it right. DIshonesty is viewed by every major religion and philosophy as wrong. Somehow our society is beginning to overlook those norms, though, and that moral foundation appears to be eroding.

Cheating on an interview is a serious disservice to the interview candidate. Generally, here's what happens... In an interview, when I sense a candidate is responding with a memorized answer, I immediately change the question. Not to something entirely new, but I morph the boundaries of the current question, just enough that a memorized response won't apply. Three things generally happen: 1) the candidate is absolutely flustered and pretty much gives up, 2) the candidate plows ahead (using the memorized answer) and totally blows the opportunity, or 3) the candidate, who actually does have a command of the topic, responds smoothly. In two of the three cases, however, that candidate is finished-there will be no 'hire' recommendation coming from me, and as hiring manager that usually means the end of the interview.

So the candidate makes themselves look bad in my eyes. But what is actually worse is the candidate who actually gets hired based on a misperception of their abilities. That candidate is generally placed into a position for which they are unprepared and underskilled. They are uncomfortable, they struggle, and ultimately end up fired because of their poor performance. Now that individual has a huge black mark on their resume!

It also hurts the company. When a candidate is hired, the company is looking to that person to further their business objectives. The failure to deliver on those objectives has a negative impact on the company bottom-line. It also takes time to fire an employee, meaning the company wastes more energy.

If you're about to begin the interview process, reading up on sample questions is fine. I bet, in fact, that reading the answers to those questions will also be OK, if it's done with the idea that the interviewee will learn the principles being probed in the question, and if they will push themselves to answer variants of the question. Digging deep and acquiring topical fluency (the ability to not just answer a question, but to deal with permutations and demonstrate an understanding of the underlying concepts being probed) is a great thing. As a matter of fact, I strongly encourage testers who want to improve their theoretical skills to do this very thing - in groups, sit down and bounce around the answers to questions like this. But don't bet your next career step on a memorized answer--in the end, no good can come of it.

What Makes a Good Test Case

2008-01-18T08:47:00.001-08:00

Technorati Tags: testing,qa,software,engineering,quality assurance

I recently answered this question on the MSDN testing forum and thought it'd be something good to post on. Here's my answer; read the conversation at http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2597561&SiteID=1&mode=1

There are probably two 'paths' to answering that - the first path examines why you're testing at all, and the second looks at how you're actually writing your cases. In other words: strategy and process (ugh, I know..).

Some organizations view and implement testing as all about QA--validating that an application fulfills certain requirements (IE a tax calculation is completed and returns the expected result, or a server can serve up X pages per minute). I've worked at a company like that--they were implementing packaged software and they only cared that it accomplished what they bought it for. (Ask me what I think about that approach...) Other organizations have to be (or choose to be) much more focused on overall quality. Not just 'will it fit my bill' but 'is it robust'. So there's a subtle difference, but the point is that a good test case is a solid step toward accomplishing the objectives. For instance, if the project is an internal line of business application for time entry, a test case which validates that two users can submit time concurrently and the data will retain integrity is a good test case. I think a case written which validates layout pixel by pixel would be a waste of time, money, and energy (would it get fixed, anyhow?).

Another point of quality for test cases is how it's written. I generally require my teams to write cases which contain the following (and I'm fine with letting them write 'titles only' and returning to flesh out later; as a matter of fact, one-time projects I generally shy away from much more than that).

Has a title which does NOT include 'path info' (ie, Setup:Windows:Installer recognizes missing Windows Installer 3.x and auto-installs). Keep the title short, sweet, and to the point.
Purpose: think of this as a mission statement. The first line of the description field explains the goal of the test case, if it's different from the title or needs to be expanded.
Justification: this is also generally included in the title or purpose, but I want each of my testers to explain why we would be spending $100, $500, or more to run this test case. Why does it matter? If they can't justify it, should they prioritize it?
Clear, concise steps "Click here, click there, enter this"
One (or more - another topic for a blog someday) clear, recognizable validation points. IE, "VALIDATION: Windows begins installing the Windows Installer v 3.1" It pretty much has to be binary; leave it to management to decide what's a gray area (ie, if a site is supposed to handle 1,000 sessions per hour, it's binary - the site handles that, or not. Management decides whether or not 750 sessions per hour is acceptable)
Prioritization: be serious... Prioritize cases appropriately. If this case failed, would this be a recall-class issue, would we add code to an update for this case, would we fix it in the next version, or would we never fix it. Yes this is a bit of a judgment call but it's a valid way of looking a the case. Another approach is to consider the priority of a bug in terms of data loss, lack of functionality, inconvenience, or 'just a dumb bug' way.
Finally, I've flip-flopped worse than John Kerry on the idea of atomic cases. Should we write a bazillion cases covering one instance of everything, or should we write one super-case. I've come up with a description which I generally have to coach my teams on during implementation. Basically, write a case which will result in one bug. So for instance, I would generally have a login success case, a case for failed log in due to invalid password, a case for failed log in due to non-existent user name, a case for expired user name or password, etc. It takes some understanding of the code, or at least an assumption about the implementation. Again, use your judgment.

I read the response that a good case is one that has a high probability of finding a bug. Well... I see what the author is getting at, but I disagree with the statement if read at face-value. That implies a tester would 'filter' her case writing, probing more into a developer's areas of weakness. That's not helpful. Hopefully your cases will cover the project enough that all the important bugs will be exposed, but there's no guarantee. I think the middle ground is appropriate here - a good case 1) validates required functionality (proves the app does what it should) and 2) probes areas where, if a bug is found, the bug would be fixed (in a minimal QA environment) or (in a deeper quality environment) advances produce quality significantly.

BTW: one respondent to the question replied and said a good test case is one which brings you closer to your goal. Succinct!

Hope that helps!

John O.

john@yourtestmanager.com
http://www.yourtestmanager.com
http://thoughtsonqa.blogspot.com

How Can I Become a Better Tester? Part IV: Mentors

2008-01-18T08:39:00.001-08:00

Technorati Tags: testing,qa,quality assurance,software,engineering

A major step for career growth, in any role, is to find a mentor. Mentoring relationships come in many forms, the most common of which are:

Working for someone
Working with someone
Having a formal mentoring relationship
Having an information mentoring relationship
Reading and participating in specific test focus groups

Working With Someone

The best way to learn from someone is to work with them, elbow to elbow. There's no better way to learn. I think of a couple of great test managers I had when I was a test lead. Dan, a test manager in Microsoft's Mobile Devices Division, is a fantastic manager. He is great with people, protects his teams from politics and struggles above, and understands quality. One thing I learned from Dan is that, while a tester's appetite for more time or resources is never satiated, we can get the job done anyhow. When I complained to him that we shipped a product with too few people, he asked "Did it ship on time?" [Yes] "Have there been any recall class bugs?" [No] "Then you must have had the right number of people."

Another great mentor was Mike, group test manager for Live Meeting. Mike and I didn't see eye to eye on everything, but what a great manager he was! He enabled and trusted people to do way more than they may have ever done. He put me in charge of a beta release of Live Meeting (then called Placeware), and let me drive shiproom meetings of several releases of Live Meeting. He also knew how to have a lot of fun on the job. I miss many aspects of working with Mike, frankly. And I try to make every team member's experience the same - lots of opportunity to grow, lots of fun, and high expectations.

I'll make a wild statement: in your first few years of your career (generally two to three) WHO you work for matters significantly more than HOW MUCH you make. The first years of your career establish a foundation which will determine how quickly you will grow and what kind of habits you will form. If you're fresh out of college and have a choice between working at lower pay for an incredible lead, or earning more and working at a 'code factory', may I recommend you take the former? Establish yourself early on in your career, learn the principles, and THEN go out where you can impact and be rewarded accordingly.

Working With Someone

Almost as good as working for a great test lead is working with a great tester. At any level. In Microsoft as a junior tester, I worked side-by-side with a person who became a great friend. He taught me about equivalence classes, boundaries, and other key concepts. He showed me how to fight for bug fixes. He showed me where the bar should be!

Twelve years later, I was managing a team of 100. I worked with three test leads who taught me a lot. Sri taught me about getting the job done - he just sticks to the job until it's done (I've always prided myself in being known as a person who gets the job done, but Sri showed me how to take this to the next level). Debbie taught me about putting your head down and working through challenges - stick-to-it-ness, if you will. And Jenn taught me about taking pride in whatever you do. We never stop learning, especially by example.

My current manager is much the same. I have never worked for anyone as diplomatic as Tony (and I don't mean that in an office-diplomacy, fake-smile way). He really cares about people, how they feel, and what they think about their job. He's also ready at any moment to take advantage of a teaching/mentoring opportunity.

Having a Formal Mentoring Relationship

At Microsoft, each new hire had a new-hire mentor for their first three to six months. This mentor was the go-to for pretty much anything, from "where's the printer?" to "do you think I'm making my goals?" After that, everyone was advised to find a mentor within the company, and Microsoft even has an internal site dedicated to finding and maintaining a networking relationship.

The same should be true for you. If you're new to a company, I recommend you find yourself an internal person to mentor you. Once you're established, look around and find yourself a mentor. Generally you want to find someone who's ahead of you in some way (technical, project management, leadership skills, etc.). Ask them to be a mentor and be very protective of the time you take from them - make sure every time you meet you have a productive conversation. Never ask them to do your work - ask for help reviewing what you might think is the right proposal, and get feedback into specifics.

Have an Informal Mentoring Relationship

A key requirement for me is to work with great people from whom I can learn. As I pointed out, I learn from people I manage. I try to learn in almost every circumstance and every person. If there's someone you learn from a lot, try to be around them a lot even if you're not in a formal mentoring relationship.

Participate in Forums, Groups, Discussions and Seminars

Finally, there's a lot to be learned by reading and participating in forums, groups, discussions and seminars. I'm active on several forums (MSDNs testing discussion forum, agile testing forum in Yahoo Groups, etc.) I learn a lot just by lurking (although I'm so opinionated that it's impossible for me to lurk for long) or by joining in on the conversation. Other places to learn include seminars, networking groups, and even podcasts (BTW: stay tuned for a podcast from me over on http://www.searchsoftwarequality.com, where I'm a Testing Expert)

How Can I Become a Better Tester? Part III: Going Beyond Stated Requirements

2008-01-18T07:56:00.001-08:00

So you've spent some tme becoming more aware of quality and what quality means. You've been looking at the differences between a Mercedes and a Hyundai. You've also started reading up on quality and on engineering. Great start! What's next? Well, the next step is to realize you need to look beyond the stated requirements and dig deeper.

In my opinion, functional or business requirements docs are like nets. They catch a lot of 'big stuff' but they can easily let little--albeit important--stuff through. For instance, a business requirement might be that a tax calculation be performed on a per-item basis. t the same time, it might not mention that the overall tax rate, which is rounded to the lowest cent, needs to be calculated on the total purchase and not the individual items.

Disclaimer

I have no idea what the rules are for tax calculation. I don't test it, never have. And even if I did know, it'd only be specific for the US. So please - look over the details in this example and try to recognize the key point...

End Disclaimer

The Tax Man Cometh

As an example, assume a user buys one $0.55 candy bar, a $1.75 bag of chips, and a $4.27 bottle of antacid. At 10% tax, that would look something like this:

Item	Cost	Tax	Total
Candy bar	$0.55	$0.055	$0.60
Chips	$1.75	$0.175	$1.92
Antacid	$4.27	$0.427	$4.69
			7.21

Tax is calculated on each item, and in all cases is rounded down to the lower cent.

However, if you factor tax on the sub-total of the items, it may actually be more! The subtotal of these three items is $6.57, and tax is $0.65. Calculated this way, the total is actually $7.22.

OK - I agree. This is a case of a missed business requirement. But this is a great example of how a tester needs to be on his or her toes, asking questions and always exploring for what can go wrong.

Not Another Bad Lego!

I'll take a manufacturing flaw as another example, which hopefully you can extrapolate to software quality. I have three sons, spanning just 4 years, and as a family (Mom is the ringleader here), we are Lego entusiasts. Birthday presents are almost always Legos - these days, they are super-complex Star Wars sets. Whenever my boys have saved up $10 or so, we're off to the store to buy a new Lego. Several times in the past couple months, we have picked up Legos with one of two problems: 1) missing peices and 2) poor fit. There's nothing more frustrating for a kid than to have their new Lego missing a critical piece--or for the parent who has to drive 20 miles to return the Lego!

I'm sure there's a manufacturing QA requirement here that each individual bag of pieces has a certain total weight. That's how they make sure all the pieces get into the box (how they miss a piece on this test is beyond me). Two days ago, my youngest bought himself a new Bionicle, only to find that the Bionicle's shoulder-claw (you have to see one to understand) was in the box but it didn't stay in place. Turns out, a manufacturing flaw caused the shoulder claw's insert tab to be too small, producing insufficient friction for it to stay.

So as a QA engineer at Lego, I'm sure I'd be dealing with the requirement to have all the parts in the box. But would I be asking about how we ensure EACH part shipped in EACH package has the right fit? That's going beyond the requirements.

Hey, You Stole My Car (Analogy)

OK, one last analogy and I think the point will be well-clarified. Let's pretend you are responsible for quality for an automobile. The requirements are that it be capable of a certain miles per gallon (KMPL), that it run for so many successive hours, and that it fit so many passengers. So let's assume these are 25 MPG, a service life of 100,000 operating hours, and it seats 4.

In the early 80's, AMC made a car called the Pacer. It was small, rather fuel efficient, and cheap. It sat five (although we had many more than that in my friend's Pacer once). Service hours? My friend Kate couldn't kill it, no matter how hard she tried - she drove it for a year without putting oil in it!

A colleague of mine at Microsoft recently had to buy a car. He chose a VW Toureg V10 diesel. He actually hits 30, 35 MPG. The car has a proven service life well in excess of 100,000 hours and it fits 5 (very comfortably, I might add). Which car is of higher quality? Why?

I'll leave you with that thought. Dig deeper, go beyond the stated requirements. It's the difference between an Ambassador and a Skoda, a Pacer and a Toureg.

How Can I Become a Better Tester? Part II: Read

2007-12-27T20:26:00.001-08:00

OK - so you are paying more attention to quality now, right? Starting to ask more about what makes a Mercedes so much better than, say, an Opel or a Dodge. What's another step you can take to become a better tester? Simple - expand your horizons by reading. That's what this blog entry is about: what can you read?

The first place to start comes for free: read off the Internet. Read blogs (http://www.testingreflections.com is a great one--there are many out there). Also, a site I'm affiliated with (http://www.searchsoftwarequality.com) is a great resource. There is a performance test expert who frequently contributes, as well as several experienced test managers.
Related to blogs, start reading test forums. I have found that the Yahoo Agile Testing Group is a great source of information. Another good source is MSDN's forum on software testing.
The next place is to read books on testing. Cem Kaner, James Bach, James Whittaker are all great authors with significant experience. Whittaker has a series of "How to Break..." books which is rock-solid with good day-to-day steps for your testing.
The next great source of reading is books on the topic of engineering. These books talk about best-practices in teh engineering discipline and always lead me to think about how my team is approaching a project or a problem. Some of my best test strategies have come as I've read development manuals such as:

Test Driven Development with Microsoft .NET (Newkirk and Vorontsov)
The Object-Oriented Thought Process (Weisfeld)
Software Project Survival Guide (McConnell)
Writing Secure Code (Howard)

Conferences and seminars: let's face it, a lot of conferences are actually just about marketing something. But even those conferences offer the opportunity to sit with experienced testers and learn from them - either in presentations or during the 'chit-chat' portion of the conventions. A couple of thousand dollars is a lot of money to spend on a conference, but sometimes it'll pay back in spades, just with the new network of associates you build who can help you through a challenge you may be facing. I'm speaking in May 2008 at PSQT's Vegas conference, if you're interesting in meeting me face-to-face.
Certifications: OK - to be clear, certification is NOT a substitute for experience. It alone means nothing, but a certification can often lead you in the right direction in improving your skills. It can build a framework for you to 'flesh out' with experience over time. PSQT offers certification and there are a few other courses available. Again, I stress: certification in and of itself can only teach you the process and steps recognized as historically good testing. It's not everything you need to be a great tester, but it is a good framework.
Practice: the best teacher is practice, and that's what you really need. Be really weird - start a testing group which meets after hours and just tackles weird testing problems (meet at a local cafe and talk about how you'd test the espresso machine or the retail point of sale system).

These are just a few ideas. There are many, many ways to learn. The key to learning, however, is wanting to learn - and for the right reasons. If you want to be a better tester so you can move up the food chain and become a better manager, well, you'd better think about a career as a project manager instead. If you want to be a better tester and that's it, trust me - you'll learn. And the growth opportunities will present themselves, too.

Next subject: going beyond the requirements.

How Can I Become a Better Tester?

2007-12-15T08:54:00.001-08:00

In all my blogging and participation, I have met many testers. One recently asked me "How can I become a better tester?" As I answered, I thought "Hmm - might be a good blog series!" So here we go. Most of this is aimed at beginning testers (1-3 years) although I have to admit I would benefit from some touch-up work in each of these areas!

I gave my friend four steps to take. I'm sure I'll come up with more, but let's look at these four briefly, and then I'll spend a blog or two on each as time goes by (hopefully I'll finish the last one sometime before the Rose Bowl).

Become aware of quality - what is it, what does it look like, what does it feel like?
Read - read everything you can get your hands out about quality. Start off with books about testing. I really don't even care about what testing methodology (agile, monolithic beastly projects, etc.), just read.
Related somewhat to (1), recognize that quality means going beyond functional and business requirements. You need to think of yourself as the gateway to quality or, in an agile world, as the trustee for quality. Teach people about quality and manage up for quality.
Find a great test lead, test manager, or test mentor to work with. If you can be employed on the same team, that's great. If you have to be in a less formal mentoring relationship, that's great. Learn from this person, everything you can.

First: become more conscious about quality. I actually gave my friend a homework assignment - as this person lives in India, I asked them to compare two vastly different automobiles: the Ambassador and the Skoda. Ironically, the Skoda is the butt of many a joke in Eastern Europe, but it's actually been taken over by VW/Audi and, internationally, is an incredible automobile. It competes with (and beats) Honda in India as the ultimate status car within reach of the rising middle class. The Ambassador? Well, it's truly the Indianized car--it's one of two cars mass-produced throughout India's history. When you compare the quality of the two, well, it's like comparing a Chevette to a Lexus in the US. Or, I suppose, a Trabbi to a Mercedes in Europe. No offense intended towards the Ambassador--it's just reality.

So I like to think about quality and compare products and services. If you live in the US, try this - go to Mervyns or Kohls to buy a suit. Then go to Nordstrom. The difference is beyond belief! If you don't think consciously about it, though, it's hard to put into words what the difference is. A good business analysis would miss the point either - consider the business requirements:

Stock on hand: variety of sizes, multiples of the most common sizes. Mervyns: check. Nordstrom: check.
Merchandise is on the rack, readily visible to the customer: Mervyns: check. Nordstrom: check.
Sales reps available to answer questions: Mervyns: check (you do have to seek them out). Nordstrom: check.
Someone who can ring up the transaction: Mervyns: check. Nordstrom: check.
Merchandise is priced right: Mervyns: check. Nordstrom: check.

A tester who focuses on only the business requirements here would say both companies are of equal quality. But think deeper - look at the experience in both situations. The Nordstrom experience is full of stuff that could be measured, but it's necessarily the first thing you think of. How friendly and helpful is the service? What is the atmosphere like? What's the merchandise like? Will it feel cool during summer and warm in winter? Is it scratchy? Is it comfortable overall?

So the purpose of the homework is to think deeper about quality. Is quality just meeting the requirements, or is there more? Is it about understanding the customer and knowing there are often unwritten requirements?

How would this apply in the real world? Well, I've already blogged about a huge data corruption issue that arose at a previous employer because the consulting firm running an implementation there insisted negative testing wasn't necessary. They focused (barely) on proving the software met requirements--since none of the requirements discussed issues like "the data needs to have integrity" or "the client must respond to and handle error messages from the server", no testing was done in that area. Thirteen million corrupt rows later, four workstreams came to a halt while the data team narrowed the issue and implemented a fix - oh, and fixed the corrupt rows.

Becoming aware of quality is the first step to contributing to quality. It's what helps us lift our eyes beyond requirements and to be stewards of quality.

Technorati Tags: QA,Quality Assurance,Software Testing,Testing,Quality

What's the Right Tool?

2007-12-12T15:46:00.001-08:00

I am pretty active on several testing forums (Microsoft's MSDN forum, and a Yahoo! group for agile testing are the two I frequent the most). I cannot count the number of times someone sent an e-mail asking which big, monolithic tool is the right one for them.

My friend (and manager) has a great response to this. At the LDS Church, we are frequently asked by new testers "When are we going to standardize on an <insert category here - performance, automation, etc.> tool? His response: you're engineers. Look at the tools in use in the organization today, look at the tools available in the industry, and make the best decision for the organization. Sometimes you might give up a little functionality or ease of use in exchange for a tool which is already used in-house. You get a built-in support forum, and sometimes you can even leverage existing tests. Other times, you'll probably pick a best-of-breed tool.

Our performance testing is a perfect example of this. When I came on board, we were using JMeter, an open-source java-based tool. The benefit of JMeter? It replays Apache logs. There's no better mimic of production than replaying production logs! The problem? We have been seeing weird things with JMeter - for instance, if a connection times out, JMeter doesn't drop it and move on. Instead, it opens another connection request but leaves the first request hanging. In some ad-hoc experimentation today, an engineer on my team started with just 5 connections and ended his test at 10 connections. Scale that to the 400, 500, 1000 connections we were using to load up www.lds.org, and you end up with incredibly unrealistic test scenarios! I never felt we could trust JMeter.

So I started looking around, and I have settled on WAPT (Web Application Performance Test Tool, http://www.loadtestingtool.com). It's a commercial product, but only costs $350 per license. There's a lot I love about it - first of all, connection management seems to be so much more reliable. Secondly, the built-in reporting is incredible. Third, there are actual support personnel if there's an issue. Oh, and fourth, I can bring a $250,ooo enterprise server to its knees with a dual-proc Dell running WAPT, whereas I needed 10 machines to do the same with JMeter. I'm pleased as punch with it. However, we've had to abandon a year's worth of test scenarios and rebuild them. I've had to justify my decision to change tools. And I'm the only person in the org running WAPT (right now).

The bottom line is, there's a better way to get advice about tools. Don't ask "Which monolithic tool is the best?". Describe your test environment and the project you're working on, and ask "What are some tools people have used for this type of testing? What are pros/cons/strengths/weaknesses of these tools?" Gather information and make your own decision with advice from others.

You're the engineer on the spot and you need to make the final decision. If you're a test lead in Gurgoan, don't let me decide for you what tool to use.

BUT: don't hesitate to ask! If there's one thing most testers have in common, it's an eagerness to share their (opinionated) opinion about something. And I personally love to give advice and help out. This isn't a censor or a rant about asking questions - it's about people needing to step up and be engineers.

JTO

WWII and Test Management

2007-12-05T12:38:00.000-08:00

I've been on a WWII kick again, reading a lot about the war. Eisenhower's job was to decide on the strategy - do we attack Germany in one concentrated punch (Montgomery's plan) or do we attack along the entire front (Eisenhower, Patton's plan). Patton's job was to figure out how to implement--he had to sit and watch as Montgomery's punch plan failed in places like Arnhem. Then he got to decide strategy for crossing the Rhine and driving for the win - Berlin or the Eagle's Lair? Once these decisions were reached, they were carried down the chain of command - there was a big marshmallow layer of bureaucrats who procured supplies, managed replacements, etc. and then - and then there were the lieutenants.

The senior leaders (generals and majors and such) are the test managers. High-level strategy, supply mangement, personnel. They're negotiating at a high level with PM or dev on things like engineering process and such. They may be fighting over budget and headcount, too. The lieutenants? These were the boots-on-the-ground leaders who took initiative and got things done. They trained for D-Day for over two years, learning to climb cliffs and take out bunkers. The day after D-Day, they encountered hedgerows and had to make up a new strategy on the fly - and good thing, too, that they did it--there were there, in the heat of battle, and they knew what worked and didn't. They called in support from the Navy or Air Force. These are your test leads, the folks rallying the troops. They're the folks who should decide to implement MBT or PICT or the likes. They're the ones who need a great working relationship with dev leads (something I'm not doing great at right now myself - need to work on that). They're making sure the privates are moving along, they're training new leaders, and they're calling for sacrifice at appropriate times.

Black Box Testing - time to go?

2007-12-05T10:11:00.000-08:00

Regarding black box testers... Having lived in both product software AND IT engineering worlds, I'm beginning to understand the root cause of the argument about the value of blackbox testing. Developing user product software requires a ton of testing; at MS, we did this against more functional specification rather than business requirements. I'm convinced software applications (even apps like Oracle Retail, which suffer significantly from the lack of blackbox testing) need serious blackbox testing because the target audience and the way the product will be used are so varied. Integrating business applications in an IT setting, however, really doesn't need as much and yes - I believe this could be covered by business personnel rather than by QA engineers.

But here's the conundrum... Until the IT industry accepts testers as engineers and allows them to work with dev as code is developed (and this on a day-to-day norm rather than an exception to the rule), it'll be difficult to convince engineers to go into testing and to stay there. In my role as TM for Circuit City's MST project, I interviewed 30-40 people for open roles and found practically no one who could code. None of my 30-member team in Richmond could code beyond writing XML (well, one could--he was my best test lead, too). Very few of the personnel provided by IBM, the engagement partner, could code either. One tester they provided, who had a rather high bill rate, was justified b/c she had 'retail experience'. She worked in a clothing store in NY during college!

So there's a chicken/egg thing which has to be solved one company at a time. Convincing management to 'up the bar' for test engineers, convincing development to be more agile, etc. are all necessary steps and I think it's still going to take a while. I for one am totally convinced that you get more for your buck with whitebox and gray box testing, and that you get way more for your buck hiring technical testers. But convincing a CS grad that she should consider test when all the glamour is associated with development? That's a challenge.

All that having been said, don't scoff blackbox testing. A great test engineer (very rare commodity) is a good programmer who excels at white box testing, who can black box test as well. Those finishing touches, esp in product development, are critical to project success.

Shameless Plug

2007-10-16T18:08:00.000-07:00

Looking for two-way interactive information on software quality? Head on over to http://www.searchsoftwarequality.com and check it out. There are articles, webcasts, and even an 'ask the experts' column. Yeah, I am an expert there... Come on by and ask a question!

JT

Testing Patterns and Practices

2007-10-09T21:01:00.000-07:00

I was chatting with a good friend the other day, and he commented that he thought it'd be logical to have testing patterns and practices. Well, that opened up a whole line of thinking, and I decided to post a blog on it and see what comes up!

So sound off - what patterns and practices do you think are logical for testing? I can think of a ton--both programmatic and just sheer test cases. For instance:

Test Cases

Boundary conditions (int): lower boundary, lower - 1, lower + 1; upper boundary, upper +1, upper -1
Text/string: alpa, numeric, 'extended' (takes you back a few years, doesn't it?), double-byte, multi-byte, UTF8
Input field: valid input type, valid input length, invalid input type, invalid length, XSS/SQL injection

Code

Web service input, output
Accessing fields on a form in an application
Web request/response (threaded)
...

This is short list just to get people thinking. Time to get those creative juices flowing folks!

Where've I Been?

2007-10-03T23:51:00.001-07:00

So I know I've been somewhat inconsistent in posting lately. It's one of those things - every job has had a push time, and this is one for us. The LDS Church holds a world-wide general conference every 6 months - leaders from the Church meet and give sermons twice a day, two hours per session, Saturday and Sunday. For http://www.lds.org, this is the web equivalent to Christmas and Easter combined.

Coming into the October conference (this weekend), we've been working through a series of stability issues. That means pulling some all-nighters (literally - I came home yesterday as my son was getting up for school) to drive load when the site isn't utilized.

Test has been involved primarily as helping generate test load against the servers. It's an interesting challenge - who ultimately owns 'quality' on a web site? Test? Operations? Test is responsible for the applications coded, whereas Ops is responsible for the configuration and maintenance of the servers. The good news is, at the Church we all pitch in and work together to get through this kind of thing.

So that's where I've been. Hope to get back to my posting about quality - what is it, how does it happen - very soon. Meanwhile, catch some rest for me too, OK?

Exactly...

2007-10-03T23:43:00.000-07:00

I was catching up on some Google alerts and came across an article on ZDNet:

40 IT failures caused by software bugs by ZDNet's Michael Krigsman -- Rick Hower, who runs the Software QA Test Resource Center has compiled a lengthy listing of “major computer system failures caused by software bugs” Here are several entries from that list: A September 2006 news report indicated problems with software utilized in a state government’s primary election, resulting in periodic unexpected rebooting of voter check-in machines, [...]

This article is a great source of justification. I've had my own negative experiences caused by inadequate testing (I keep harping on it, but 13 million corrupted rows in a production database is a bad thing - before my time, so I don't bear responsibility, but I failed in several related arguments about the level of testing needed).

This goes back to my post about levels of risk. If I'm an online web site, I need to devote more resources to my online transactions and account privacy than I do to my site UI. Yes, my site is my 'best foot forward' but all the benefit of a well-designed, well-implemented, well-tested site goes out the window the instant I'm hacked.

Business leaders just don't seem to get it... Testing is NOT about proving functionality. That's what IBM and Circuit City management kept pushing us to do. I had a 2-hour argument with an IBM dev lead my first week at Circuit over this very topic - I was trying to do too much negative testing (one case was too much, in his eyes).

Finding that leadership to influence from 'below' is a challenge. It takes patience and consistency--constantly reinforcing the value of testing. Articles like this one from ZDNet help, too. Unfortunately, many businesses refuse to learn from others' experience. Wisdom or Experience - we can learn from one or the other. Experience is truly an expensive teacher though!

Trackback URL: http://blogs.zdnet.com/projectfailures/wp-trackback.php?p=427

How much is enough testing?

2007-09-26T23:11:00.000-07:00

Amazing article! Today hacker Robert Moore spoke up about how he pulled off his crimes (hacking into tons of VOIP serves and reselling communications services). His way in? So simple - he just used the default password for common communications devices (Cisco routers, etc.). Once in, he took control and routed traffic as he desired. URL: http://www.informationweek.com/news/showArticle.jhtml;jsessionid=WIARC3KZRXVXQQSNDLRCKHSCJUNN2JVN?articleID=202101781&pgno=2&queryText

This blog isn't about the hack. It's elegant, but in the end common petty thievery and nothing worth a bit of praise. This blog is about the quote from page two:

"Kenneth van Wyk, principal consultant with KRvW Associates, said leaving default passwords up is a widespread and dangerous problem. "It's a huge problem, but it's a problem
the IT industry has known about for at least two decades and we haven't made
much progress in fixing it," said van Wyk. "People focus on functionality when they're setting up a system. Does the thing work? Yes. Fine, move on. They don't spend the time doing the housework and cleaning things up." "

How many times have I been told in the past year "Just run through the test cases" and "just test the positive cases"? I was literally told by one employer (not my current) that SQL injection and other user-security cases were unimportant. This was from an employer going through multiple rounds of lay-offs and terrible morale.

Testing is about proving things work but it is about so much more. If a web page is served up, does it mean it 'works'? What if it takes 3 minutes to serve up the page? Is it OK then? If I can update information about an entry in my database, but I'm not monitoring for errors, can I say it works? What if, each time an update is sent, the update is 'written' but an error is thrown? If all I'm looking for is a row with the updated information, and I'm not running a negative test case to ensure the old information is gone, can I say it worked?

Now that the recruiting series is behind me, I'm going to spend some time investigating this concept. When is testing REALLY finished? (And I mean that as the verb 'testing', not the noun "Testing".)

Recruiting for Intellect

2007-09-26T22:27:00.000-07:00

One other skill I am looking for during an interview is intellectual horsepower. This is probably the most controversial element of interviewing – for a kick, read How Would You Move Mount Fuji to learn more about the subject, and for a ton of great interview questions!

The basic argument against the puzzle question is that it’s generally an ah-hah question and doesn’t show whether a candidate can perform the work they’re being interviewed for. OK – I buy that… I could bring in a Nobel-winning primatologist and they’d probably crush my interview question but they’d be a terrible developer. However, when I’m hiring I’m not just interviewing for someone who can slap together a few DHTML objects… I’m looking for an engineer who can code, for sure, but who can pick apart problems and drive creative solutions to them. I’m not just looking for someone who can come in and code in Java today – I may need a low-level C++ driver written tomorrow or I may need someone who can build a replacement to a multi-million dollar line-of-business application. So you know, I agree—no problem question will allow a candidate to prove their coding ability. But I also need to see their problem-solving ability.

That having been said, there’s more than one way to evaluate what my former General Manager calls IHP. As a matter of fact, I quite often couch my question in a coding question. For instance, I may throw a quick and dirty pointer question at a candidate (to see the depth of their CS skills—and you’d be amazed at how few candidates can code with pointers… it’s a shame), and then I’ll follow it up with a question like “Design a tool which takes in two string arrays – one is a list of words, the other is a crossword puzzle—and evaluates whether all the words are in the puzzle.” That’s an engineering challenge – seems straightforward, but it’s actually a puzzle.

I am not afraid to throw the puzzle question at people either. Yes, most of them are ah-hah questions where there’s only one answer and you either get it or you don’t. I try to stay away from them, although I don’t mind them too much. Very much like my coding or testing question, I’m less concerned with candidates arriving at the answer and more concerned about them showing critical thinking abilities, the ability to step back and re-evaluate a solution, and a little bit of thinking outside of the box.

What are some sample questions? How Would You Move Mount Fuji is a great source for these. Some I like are the bridge over the chasm (four people need to get across the bridge, it’s dark, and you only have one flashlight—how fast can you get them all across?) or the miner stealing an ounce of gold for every pound. There are thousands of these questions though. And anyone who thinks they aren’t fair should know that I actually run them past my Boy Scouts and they can solve them (sometimes with a little hand-holding, I’ll admit – but these are 12-year old boys!).
In evaluating, be sure to avoid the emotional evaluation. Don’t be overjoyed or think the candidate is a hire just because she got the answer. Stop and think about HOW they got there. Did the candidate just stumble on the answer? Did they guess? Did they get there so fast that it’s obvious they knew the question in advance?

Look for critical skills. Can they break the problem down? Do they arrive at an initial answer, but then continue to probe looking for a more elegant solution? Are they organized in their approach? Are they cool under the pressure? (That’s a lot of it for me – I am looking for candidates who can tackle a tough problem—with a senior manager in the room.)

Test scores and higher ed grades are a good indicator, as well, of performance. Keep an eye out, though, for the brainiac with no common sense. You probably don’t want them unless you have some really deep experimental research type of effort (cryptography?).

In the end, I’m looking for a balanced candidate. A super-smart person with poor coding skills isn’t as helpful to me. A hard-working coder who can’t think his way out of any problem might help me short-term with the challenge I’m facing at the moment, but I’m not convinced they’re a good hire. They might make a great contractor, but I’m only looking to hire bright people who will grow with my challenges and needs.

Recruiting for Potential

2007-09-05T23:36:00.000-07:00

Potential. OK I admit it, I left potential out as one of the hard characteristics to probe for. What is potential, anyhow? Well, one definition is that it’s the ability for a candidate to progress to higher levels of contribution within your organization. If hired in as an entry-level test engineer, could the candidate progress to a technical lead? Can she become a test lead, test manager, or even an engineering director?

So how do you gauge this? It’s not as straightforward as measuring, say, technical skills or the ability to tear apart an application into test questions. But there are some guidelines I like to use in my recruiting which might help you.

First of all, I’m looking for a level of maturity about career opportunity. I don’t expect a college candidate to be as realistic or mature about where they want their career to go as I would, say, a five-year veteran. However, I’m still looking for a candidate who wants to progress and who wants to make a difference. I’m looking for a candidate who’s evaluating my opportunity on the basis of where she can go next and what skills she can gain. I’m also looking for a candidate who’s thinking about how they can contribute. A campus hire isn’t necessarily going to change the way my department does everything, but they may well come in with new coding skills or a process improvement they picked up during an internship.

Next thing to think about is what have they demonstrated in the past in the area of improving their abilities? I will ask things like “What technologies have you learned recently?” or “What have you had to come up to speed on in your workplace?” For me, I’m always needing to learn something. After 11 years at Microsoft , I am facing a huge wall of IT skills now that I’m in an Oracle and IBM heavy environment (more on that someday…). So I’m learning all about open source test tools, getting up to speed on Oracle SQL and I’ll probably have to jump into Java programming (sigh). But I do what it takes to keep on top of the technologies behind my projects—I’m a professional.

There’s something next which is really tough to describe, let alone measure in a candidate. Put simply, does the candidate ‘get it’? Do they understand the role of IT in the org they’re interviewing for? Do they understand issues or challenges they may have faced (or that they face) in previous positions? Are they all blame and no responsibility? Are they open and introspective about their failures, or do they continue to point the finger at outside influences? Do they show the mental maturity to make decisions on a team or group-wide level?
As not all candidates want management to be in their future, you also need to probe for deeper technical skills. Do they have any experience in distributed programming environments? Do they understand that their projects have broad impact? Can they see, for example, how an automation framework they worked on in a previous team could have been used across their entire organization?

It’s a challenge to measure a candidate’s ability to grow, but hopefully each candidate is going to be with your group for a very long time, so you want to be sure you’re hiring people who can grow over the long term and won’t just consume oxygen.

Recruiting for Testing Skills

2007-09-05T18:50:00.000-07:00

I’m a test manager, so ultimately my interviewing is to find candidates who can test – who can find bugs, prevent bugs, and write tools to help in those processes. Testers have a special mentality—seriously, I’ve interviewed (and worked with) plenty of solid development engineers who couldn’t test their way out of a paper bag. It’s a special person who can code AND test. The good news is, finding a person with a ‘break it’ mentality is pretty easy. Probably the easiest of all the characteristics I interview for.

The best tool I’ve found for this is to simply throw the candidate a test question. It can be something theoretical, or it can have a real-world application. One of my favorites is to test a function which receives three integers representing the three sides of a triangle, and returns whether the triangle is equilateral, isosceles, a triangle, or invalid. This question works really well; it presents the candidate with a real world situation (who remembers what an isosceles triangle is anymore?), it has a ton of effective cases, and it can be easily moved into additional problems such as becoming a web service or the likes.

What’s the right answer? Well obviously candidates are going to have to catch the positive cases of each type. But from there the sky’s the limit—I’m not going to list all the cases here but suffice it to say that I was once interviewed by a test manager and given this question. We went on for a good 30 minutes and could have continued. And that’s one of the key points I’m looking for – does the candidate throw out a series of cases over a few minutes and give up, or does she keep going? A real tester should be able to run on and on and on with new cases, if your question lends itself to this. That’s why the question is so important.

So I’m looking for my candidates to just keep generating test cases. I’m looking to make sure they cover the basics – positive cases, boundary cases, etc. I’m going to keep an eye out for candidates who aren’t equivalence-classing their cases. A candidate who tells me 1,1,1 2,2,2 3,3,3 and 5,5,5 isn’t getting anywhere; there’s no difference in any of these cases (well, maybe 1,1,1). So rambling along with repetitive cases is no help. But going on and on and on with cases is definitely a key.

Another thing to look for – is the candidate organized? I’ve interviewed with a number of leads and managers who claim no candidate is sufficient who isn’t organized in their approach. I disagree with this somewhat—a candidate who, through free associate of thought, jumps from performance to security to positive test cases isn’t necessarily incompetent. As long as candidates 1) cover all those case types and 2) make sure they get the required coverage, that’s all I’m looking for. I happen to be rather random in my own test case generation. I may use an outline, but I’m plugging cases in all over the outline rather than working top to bottom. It’s simply how I think. Don’t mistake the trigger of association for disorganization.

Another thing I’m looking for is the candidate’s ability to think like a user. In testing a problem like the triangle evaluation, are they coming up with good use cases and showing customer empathy? I consider myself a technical guy and I’m always looking for good perf bugs or bad data manipulation or the likes. But the most recent bug I wrote up (just yesterday) had to do with how difficult it is to perform a certain workflow procedure in our line of business application. Nah—it’s not engineering. It’s about solving the problem the right way for the customer. In our organization, the IT department has made huge progress but still has room to go in winning over customers. Many groups are still working their own solutions rather than using our expertise. Fixing weird useage scenario bugs like this has an amazing positive effect on the department’s reputation for being client-centric.

So don’t just focus on whether the candidate got the obvious cases. Look deeper and evaluation for thoroughness, for customer empathy, and for the ability to derive great cases out of what seems like a straightforward and obvious tool. Your department will benefit greatly from this kind of test resource!

Recruiting for Skills

2007-09-04T23:14:00.000-07:00

One of the easiest characteristics to recruit for is hard skills. At the end of the day, I need a person to code or test—since that’s what they’ll be doing all day, there’s no better way to probe than to have candidates actually code. I was a bit surprised during my recruiting in India to see how many companies administer a coding question in the form of a seated, written test—there’s probably some value in that, because it does allow the candidate to prove their skills but a written test is such a sterile environment. When I probe for coding, I’m looking for a couple of things at once: can a candidate break the problem down appropriate, can they get the right logic, are they looking to refactor or improve their code, and are they thinking about performance or scalability. In a written test, the candidate does all that thought in a vacuum – all I get is the final code. So a written test can be used as an initial weeding out, but don’t rely on it as the sole measuring stick.

In an interview, I’m dealing with limited time—usually 30 to 60 minutes. Because I don’t have all day to get through a question, I need to pick something that will let candidates demonstrate as many skills as possible. I have, therefore, developed a few standard requirements. First, if I’m interviewing for C or C++ developers, my questions do not allow candidates to use outside libraries. I once had a candidate yell at me – literally – about that restriction, but I don’t care. Anyone can consume a library – but can they write clean, elegant code on their own? I often ask them to implement a library—for instance, one question I like to ask is reversing the characters in a string (I usually throw a twist in that, such as reversing the characters in each word while preserving word order). There are some very simple bars to measure by – does the candidate use pointers or temp arrays? Memory isn’t constrained any longer, but a clean coder is still concerned. How complex are their logic statements? Are they nesting double and triple negatives together? Do they over-complicate things? Are they getting lost in their own algorithms? How confident do they appear? Do they tackle the problem without thinking, or do they put together an approach in their mind.

I really don’t care about the solution they ultimately arrive at (for the most part). I’m mostly concerned that they write clean code, the solve the problem, and they do it with quality. I’ll take a conscientious coder who returns to his algorithm one, twice, even thrice during an interview over the cocky application who codes quickly and sloppily and just leaves the code when it’s ‘finished’.

Recruiting for Passion

2007-09-01T21:29:00.000-07:00

As I mentioned, in my recruiting I’m looking for six key characteristics: passion for quality, skills, break-it mentality, potential, fit, and intellectual horsepower. How can you tell if a candidate has these traits, for sure?'

I think of the six attributes, passion and fit are the toughest to measure—they are soft skills and thus they’re challenging to measure. I can gauge a candidate’s coding skills by having them write code. I can probe for IHP with a few challenging puzzle questions. I can count the test cases a candidate comes up with, and I can see how far a candidate will go in their career. But knowing their true passion and measuring how well they’ll fit into my organization are much more difficult to determine. But there are a few keys I use in my interviews, and I feel I’ve been right many, many more times than I’ve been wrong.

When I look for passion, I’m looking for a candidate who’s thrilled about technology. As my current CIO pointed out recently, a passion for quality isn’t necessary a good thing – it’s putting the passion in the right place. Engineers are geeks – we get into weird things. We like to program to relax; need I say more? We’ll spend an hour arguing over the purest object-oriented method to serialize an object! That’s passion, but it doesn’t necessarily make for a good tester.
The passion I’m looking for is applied technology. Great candidates are just as excited about pure object-oriented code or simple implementations, but they want to see them put to use in a setting where technology is solving a problem. That’s the key to me – can the candidate recognize the value of technology in a given situation. I once had an acquaintance who took time from his family to head into the mountains and write haikus—then burn them. Poetry that’s never shared is a waste, and technology that has no purpose is a poor use of a department’s time and money.

Questions I like to use to probe for this passion give candidates an opportunity to talk about a problem they see and how technology can solve it. Some questions I use to get candidates talking include “What’s your favorite software product?” or “If I gave you a million dollars (or a blank check) to form a team and build a software product, what would it be?”. I’m going to pay particular attention to the problem they set out to solve, and less about how they’ll solve it.

For instance, my favorite application is Microsoft Visual Studio 2005. Look, I’m not the greatest coder in the world—I’ll admit it. But VS 2005 gets even someone like me moving quickly. The IDE environment helps me get everything done quickly – I get squigglies when I mess up, I get F1 help on pretty much any code snippet I highlight. And finally (finally!!!) I get help lining up my curly braces. The productivity contained within that application is amazing—it is to development what Word was to document processing. My productivity increases hundredfold (literally…). Pure coders will scoff at me, and that’s fine. There are a ton of engineers in this world who are highly productive with a bare bones coding environment – more power to them! But when it comes to getting the masses to get their work done, nothing can compete with VS.

Now, you can take issue with my passion for Visual Studio. But hopefully it’s a great example for passion for technology. The problem? It takes a long time to code up applications. The solution? Visual Studio, .NET frameworks, and all the other productivity tools that come with the application. There’s a clear problem and a clear solution.

What if someone offered me a blank check and told me to build any product I wanted to? It’d be a dream! The biggest opportunity (which is a key word for problem) I see for technology is small business. Enterprises have a ton of great software out there, but small business owners suffer terribly. Most solutions for their business are garage-built one-off applications which have been distributed widely. Applications are characterized as procedural spaghetti code built on proprietary technologies with little or no ability to adapt to modern technologies such as Internet technologies or handheld devices. The company that comes up with a method to apply common code and technologies to various small-biz verticals will have really solved a major problem. (BTW: if you want to talk further about this opportunity, contact me via e-mail and let’s chat – I have some answers…).

Hopefully I’m making my point clear. A candidate who goes on and on about the latest extreme programming examples or who talks about the purest object-oriented code might be a brilliant developer, but can they actually contribute in the real world? There’s no telling. But a candidate who can see how technology can be applied to day-to-day business problems is someone I want on my team.

I’ll give you a couple real world examples. I don’t remember the problem I presented him with, but I hired a tester in India for an engineering team I ran. This guy is a super-geek! I once asked him if he had any hobbies, things he does outside of work… His answer was “Yeah, I like to read up on coding on the Internet.” To be frank, I think he can code circles around most of the developers on our team, and I’m sure glad he chose test because he single-handedly took us from 15% to 90% automation on our test cases in a matter of weeks. He has a code answer for every problem we faced in testing. He’s passionate about applying his coding skills.

Another example is my SDET lead on Education Products. She is a brainiac—one of those people you’re afraid of because she’s so smart. I could hit her up with a problem we faced with testing and she’d literally build a solution in a matter of hours. It doesn’t hurt that her code looks like poetry in C#, either. She is methodical and organized in her coding and produced consistent, reliable solutions.

So the best candidates out there combine hard skills with a real-world application. Finding candidates like this to join your team makes a world of difference because they’re using their skills to solve problems – whether you’re out to make money in product software or cut costs and increased efficiency in a business, you’ll only benefit from applied technology—and hiring people with a vision for applying technology to solve real problems.

Recruiting & Hiring

2007-08-17T20:37:00.001-07:00

You know, I was thinking... In my new role in QA at the LDS Church, I'm really beating the bushes looking to hire. We have an extra pair of challenges - the position is in Salt Lake City (non-negotiable) AND candidates must be members of the LDS Church, in good standing. That adds a factor of difficulty for sure!

In my time at Microsoft, I think I probably did 300 to 400 interviews, in the form of campus screens or full-length interviews. At Circuit, I was surprised at the calibre (or lack thereof) of many of our contract test candidates, and frankly some of our full-time hires--they were nowhere near the bar I had set before. I eventually had to settle when hiring contractors, because Circuit was well down the project path before I got there and there was no time to find the best candidates. But in each organization, there were also people who really stood out. What made them so special? Is there a way to find people of that caliber in the interview process?

As test managers, what are we looking for in hires? There are about six characteristics I look for during a phone screen or an interview (and, by the way, I usually get a feel for these in the first five minutes--see the book Blink for a discussion of the split-second decision):

Passion: do they have a passion for technology, and do they have a passion for testing? I'm not necessarily looking for candidates who are total geeks and know everything there is to know about DIVX or the latest XBOX game. That's good, but I want is someone who's passionate about the technology they work for. Do they see where technology can make a company more efficient? Do they see how it can change someone's life? And I don't want a tester who's in test simply because they didn't meet the bar for development. Chances are, if this is case they aren't going to meet my bar. So I need to weed through all these candidates, to find the folks who are passionate about technology and about driving quality into tech projects.
Skills: a successful candidate has to have something really special about their skills. At Microsoft, by the end of my elevenyear the company was only hiring 'developers' into test roles. The argument was that a team which was full of automators would be more efficient than a team of interactive or UI testers. Eh... not sold, personally. If there is any absolute in technology, it's that there are no absolutes! So I'm looking for someone who's going to have something great about them - maybe they are bug machines, focusing on interactive testing but simply tearing up the application they're working on. Or they might be great coders, able to solve big tech challenges. Whichever - my requirement is that they have something they are great at. Oh - and it has to match my team's needs... right now, I have an incredible interactive tester who can script automate much of his testing. I'm really hurting for that incredible developer who can test as well.
Break-it mentality: no bones about it - the candidate has to prove to me in an interview that she can take a sample application and test the snot out of it. What's a break it mentality? Well, I'll give you an example... I would present candidates with varyious test questions when I was recruiting in India. The candidates who I felt would make (at least) good testers were the ones who went on and on and on and on generating test cases. I really didn't care what the sample question was, and it never really matters. As long as the question is sufficiently complex that it has more than 20 or so cases, and as long as the candidate just spews out cases non-stop, you'll know. For the record, I interviewed 175 engineers in India, and hired 25 (dev + test). Finding *great* engineers is a challenge, no matter where you go.
Potential: I won't make a hire if I don't think the candidate is going anywhere. I'll probe for things like career growth or even challenge and goal setting. I may hire a person who flat out tells me he never wants to be a test manager--that is, if he demonstrates to me that he's been growing in his career to-date, and he's going to continue to grow. NOTE that this matters less when I'm staffing a contractor. I'm hiring contractors to tackle and finish an immediate job...
Fit: finally, the candidate has to fit. If he or she doesn't fit on my team, well, it's a waste of everyone's time. A candidate in a poor fit sucks up management's time, peers' time (in the form of gossip and complaining) and their own time in terms of effectiveness. Co-workers are less willing to collaborate, and the square peg is left to do everything on her own.
Intellectual horsepower, problem solving, and other skills: the final area I look at is a big lump of 'soft skills'. These are things like problem solving, communications, and sheer intelligence. If you've read How Would You Move Mt. Fuji, you've been exposed to the argument that looking for these skills is a Bad Thing, that you miss good candidates that way. Phooey! If I throw a puzzle question at a candidate, believe me arriving at the answer is about the last thing I'm looking for. In a good problem-solving question, I'm giving the candidate an opportunity to show project management skills, demonstrate the ability to make trade-offs, prove they can think on their feet and that they can keep their wits about them at the same time. Critical for an entry-level QA engineer? YES!! First of all, in my teams I expect that engineer to be as outspoken as my automation lead with five years' experience. Secondly, this also shows the candidate's long-term potential. If they can't think on their feet or solve problems, they might make it through their first year, but when growth is expected they're actually going to fall flat!

Next few posts will be how I probe for each of these areas.

What's the cost of quality?

2007-08-13T23:16:00.000-07:00

What is the cost of quality? Well, it's definitely an investment and, like all investments, it can have a varying rate of return. I like to look at cost in three 'investment' sizes:

Minimal investment: like the 'day trader' who is only willing to trade with $25, or $100, there's the minimal investment philosophy. The thought here is that quality investments are high-risk or low-return investments. The less squandered, the better. This is the organization that says test should only prove the positive case--if business requirement exists for a given project, that requirement and ONLY that requirement should be tested.
Limited investment: this is an individual with limited resources who invests wisely but clearly can't invest across the board in a broad, distributed portfolio. This kind of organization scrimps in areas which may cost down the road, but those 'savings' are targeted and based on some sort of strategy. Safely limited investments might include sort-cutting the authoring of test cases, risking a lack of portability or re-usability. The cases are used to ensure the initial and subsequent pre-release test passes are completed, but are not expected to be used post-release. A new web feature which is expected to remain static going forward would be a safe limited investment, for instance.
Full investment: this is a wealthy individual with the luxury of spreading her wealth across multiple portfolios and sectors. It takes money to get money and this investor is deeply rewarded. Not many people have the options she does, though. This is the organization which invests heavily in the QA process: in-depth test case development, lots of automation, plenty of lower-priority fit-n-finish test cases, and multiple rounds of testing.

So which one is right? Well, it all depends (don't you hate that about testers? We're like lawyers--we seldom speak in absolutes!). I have a hard time ever thinking of a time when the minimal investment is appropriate -- perhaps when the project is an internal-only, proof-of-concept which will never see the light of day and NEVER be the foundation (base code) for the actual project. But when millions of dollars, and hundreds of thousands of subsequent man-hours of development are riding on your project, this is just a stupid approach. It's short-term thinking, it emphasizes savings over any consideration. It's the penny-pinching fool who buys a 4,000 SF house decorated with paper trim. It'll never last!

So the real choice is between two and three - limited and full investment. At Microsoft, I participated in projects which were years in duration (some went years beyond their original ship date - I know, I know...). These projects saw repeated upgrades on top of existing code. Millions of users bought and used the products we produced, and therefore we invested heavily. It was the right thing to do! I can't think of a single release I was ever involved in where I said "We over-tested that...". I can sure think of a number of releases where I wished I'd had more time - even if it was just to automate, to benefit future releases!

At the same time, some organizations are working on web components which (for the most part) will release once and may have one or two maintenance releases. They aren't foundational code; each compiled applet is relatively standalone. On projects which have a very small chance of being revisited, is it safe to only write brief test case descriptions? If the functions are tested thoroughly, but little investment is made in repeatability or portability, is it OK? Is it the right business decision? I think so! If a team is building foundational blocks like a content management system or the business object layer, well, they'd better be sure to spend much more time QA'ing it. But if the work is for one or two releases, I think it's OK--actually, it makes the most business sense--to cut short on the test design/documentation and focus on execution.

How about you? What do you think is the right balance, the right investment? Can't wait to hear your responses!

Another blog on quality

2007-08-13T20:56:00.000-07:00

Why do we need another blog on quality? Well, because I still don't think managers get it. Quality matters! I've worked at four different organizations now (I know - not much variety, but bear with me) and eleven different products/teams. I've seen teams where quality mattered (Microsoft Education Products Group - India), and I've seen companies where consultants convinced management that quality could be achieved on the cheap (IBM @ Circuit City). So I'm adding my voice into the fray, and I hope the arguments and discussions here will help change the tide a bit.

What are my strengths/interest points?

The business case for quality and thorough testing of IT software projects
Recruiting and hiring
Outsourcing and offshoring
The test process--getting consistent

Your comments are welcome.