What's the Best Software Testing/QA Tool?

Frequently people will post a question like "What's the best tool?". It drives me nuts sometimes! Software testing and quality assurance are definitely challenging professions, but it's not fair to depend on others to solve your challenges for you! I've answered questions like this in the agile-testing@yahoo.com group, and on the MSDN forum, and I thought it was time I brought my answers together into one blog posting. That way, I can refer people back to the posting.

MSDN Answer

Lifted from my post on MSDN's software testing forum:

An American car manufacturer once tried to make a car for all things - it was a 5-6 passenger car in three different models, each with four wheel drive AND good fuel economy. It ended up mediocre at all things. You need to beware; resist the urge to pick one monolithic tool solution for such a wide variety of testing needs. The vendors (well, NOT Microsoft, of course ) would love to have you believe their tool can do it all. And in some aspects, their software testing tools probably can. However, will that one tool solution be effective and efficient in everything you do? No.

I'm not advocating a pantheon of tools, but you need to think like an engineer more than a manager or a customer. You find the right tool for the job at hand. Over time, you'll settle down to a group of 3-4 tools and you'll stick to them.

I've never used <some tool the asker referenced> - someone else will need to comment on that product's ability to do everything you're looking for. My take? It'd be too good to be true if it actually could. And I tell my three sons all the time 'If it's too good to be true, it's probably not true'.

Be wary.

Here's how you [should] approach this as an engineer:

• List out the applications you'll be testing 6-12 months from now

• Think about the test scenarios, especially those you'll want to automate

• Ask yourself how many releases of that application/project your company will perform--will quality assurance be a repeated process, or are you testing this software just once?

• Based on that, how much automation is 'worth the investment'? Microsoft Office 2007 *might* still be running automation I write in 1997--probably not, but I know 2003 runs it AND that automation is still being run, every time there's a patch or SP released.

• Now that you know how much investment to make, look for best of breed solutions for each project. Don't focus on all-in-one solutions; just look for the best tool for the job. Use demos, read reviews, ask questions. Don't ask "Is this the right tool?" but rather "Which tool do you recommend" or "I'm considering this tool for this job - does anyone have experience using this tool to do this?"

• Once you have a short list of tools, look and see if there is commonality/overlap. You'll see patterns. It's possible Rational or another all-in-one tool will appear in the list; it's equally possible none of those tools will appear.

• If there's good overlap, ask yourself what you think the pain will be if you force a tool into a job it wasn't designed for. If you can live with the pain, go for it... If not, keep looking or open yourself up to a larger set of tools.

Hope that helps (in spite of not answering your question),

John O.

 

Commentary

OK - I searched through a number of documents and can't find my other posting on this subject, so I'll just write it once more.

1. To all testers: look before you ask. Really! If you are wondering what tool you can use for a given test type, use Live Search and look for information! Performing a minimal amount of research shows respect to the audience you're turning to for help, and can actually prevent a question now and again. QA is all about searching for product defects; apply that searching capability to answering your questions.
2. You're more likely to get help on a specific question rather than a broad question. For instance, asking "What's the right tool" won't get you much. However "I've evaluated Selenium, Selenium RC, and Watij--given my situation (describe it) what tool do you recommend?"
3. Talk about the problem you're trying to solve. So "We realized we'll be running tests over and over and over again, but our UI changes frequently. What strategies can we take...?" is a question that raises the problem and let's people know what help you're looking for.
4. Asking "what's the best tool?" is like asking "What's the best car?". If you live in Germany and drive the autobahn frequently, the best car for you is far different from the best car for someone who lives in Bombay, battles thick traffic, and drives Indian roads (notoriously rough). Be specific, give details, and look for recommendations. Software testing tools are the same way - a screaming "BMW" test tool will fall apart on a Bombay road. A right-hand drive Jaguar would be totally out of place on an American highway. A performance test tool isn't the right way to automate repeated quality assurance tests. The right tool is dependent on the project, the skill sets on the QA team, the timeline, and several other factors.
5. Solve your own problems. The Internet is an incredible tool and offers us all a ton of opportunity to not have to reinvent the wheel. But don't ask other people to do your work for you! Ask for advice. If you want someone to solve your problem, ask one of us to consult (for pay) and bring us in. We'll be glad to get the job done for you!
6. Give back: as you grow and learn, give back... Don't post a question, get your answer, and disappear. Remain an active participant in the community and 'pay it forward'.

Summary? Be specific, research before you ask, solve problems, and give back. That's how to get answers online--in a sustainable fashion.

Technorati Tags: quality assurance,QA,Software Testing