Running MSTest Standalone

It’s been a very long time since I posted on this blog – I’ve been quite busy with work as well as writing articles for http://www.searchsoftwarequality.com (head over and check out my tips and Expert Answers).

I stumbled across this today—how-to guide to running MS Test standalone. http://www.shunra.com/shunrablog/index.php/2009/04/23/running-mstest-without-visual-studio/

Full disclosure, during my last stint at Microsoft, I worked on this internally. However, I never disclosed how to do it. I do not know Shunra and I honor my non-disclosure to the fullest extent.

At the same time, I’m thrilled. MSTest is a great product and I’d really love it if MS produced a standalone version. According to this post by an MSFT employee, there’s a lot of discussion around it:

this is one of the things on a higher priorities that we are considering for a future release though this is yet being worked out as such there if no formal statement. I am pushing for this & may have a update in a couple months. http://social.msdn.microsoft.com/forums/en-US/vststest/thread/2897fb68-ef48-4941-a49e-fe8cb1b5aced/

So who knows, maybe we’ll see something soon!

Quality Assurance: automated QA software testing tools

This week, I answered a question about automated QA tools. This topic comes up frequently – my base answer is “use the simplest tool that will get the job done”. There are a lot of open-source tools, but I find that JUnit (or NUnit, if you’re in .NET) and Selenium or Watij are a great combination for testing web applications.

Read my answer here: http://searchsoftwarequality.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid92_gci1346325_tax306196,00.html

Technorati Tags: software testing,testing,software quality assurance,test automation

What's the Best Software Testing/QA Tool?

Frequently people will post a question like "What's the best tool?". It drives me nuts sometimes! Software testing and quality assurance are definitely challenging professions, but it's not fair to depend on others to solve your challenges for you! I've answered questions like this in the agile-testing@yahoo.com group, and on the MSDN forum, and I thought it was time I brought my answers together into one blog posting. That way, I can refer people back to the posting.

MSDN Answer

Lifted from my post on MSDN's software testing forum:

An American car manufacturer once tried to make a car for all things - it was a 5-6 passenger car in three different models, each with four wheel drive AND good fuel economy. It ended up mediocre at all things. You need to beware; resist the urge to pick one monolithic tool solution for such a wide variety of testing needs. The vendors (well, NOT Microsoft, of course ) would love to have you believe their tool can do it all. And in some aspects, their software testing tools probably can. However, will that one tool solution be effective and efficient in everything you do? No.

I'm not advocating a pantheon of tools, but you need to think like an engineer more than a manager or a customer. You find the right tool for the job at hand. Over time, you'll settle down to a group of 3-4 tools and you'll stick to them.

I've never used <some tool the asker referenced> - someone else will need to comment on that product's ability to do everything you're looking for. My take? It'd be too good to be true if it actually could. And I tell my three sons all the time 'If it's too good to be true, it's probably not true'.

Be wary.

Here's how you [should] approach this as an engineer:

• List out the applications you'll be testing 6-12 months from now

• Think about the test scenarios, especially those you'll want to automate

• Ask yourself how many releases of that application/project your company will perform--will quality assurance be a repeated process, or are you testing this software just once?

• Based on that, how much automation is 'worth the investment'? Microsoft Office 2007 *might* still be running automation I write in 1997--probably not, but I know 2003 runs it AND that automation is still being run, every time there's a patch or SP released.

• Now that you know how much investment to make, look for best of breed solutions for each project. Don't focus on all-in-one solutions; just look for the best tool for the job. Use demos, read reviews, ask questions. Don't ask "Is this the right tool?" but rather "Which tool do you recommend" or "I'm considering this tool for this job - does anyone have experience using this tool to do this?"

• Once you have a short list of tools, look and see if there is commonality/overlap. You'll see patterns. It's possible Rational or another all-in-one tool will appear in the list; it's equally possible none of those tools will appear.

• If there's good overlap, ask yourself what you think the pain will be if you force a tool into a job it wasn't designed for. If you can live with the pain, go for it... If not, keep looking or open yourself up to a larger set of tools.

Hope that helps (in spite of not answering your question),

John O.

 

Commentary

OK - I searched through a number of documents and can't find my other posting on this subject, so I'll just write it once more.

1. To all testers: look before you ask. Really! If you are wondering what tool you can use for a given test type, use Live Search and look for information! Performing a minimal amount of research shows respect to the audience you're turning to for help, and can actually prevent a question now and again. QA is all about searching for product defects; apply that searching capability to answering your questions.
2. You're more likely to get help on a specific question rather than a broad question. For instance, asking "What's the right tool" won't get you much. However "I've evaluated Selenium, Selenium RC, and Watij--given my situation (describe it) what tool do you recommend?"
3. Talk about the problem you're trying to solve. So "We realized we'll be running tests over and over and over again, but our UI changes frequently. What strategies can we take...?" is a question that raises the problem and let's people know what help you're looking for.
4. Asking "what's the best tool?" is like asking "What's the best car?". If you live in Germany and drive the autobahn frequently, the best car for you is far different from the best car for someone who lives in Bombay, battles thick traffic, and drives Indian roads (notoriously rough). Be specific, give details, and look for recommendations. Software testing tools are the same way - a screaming "BMW" test tool will fall apart on a Bombay road. A right-hand drive Jaguar would be totally out of place on an American highway. A performance test tool isn't the right way to automate repeated quality assurance tests. The right tool is dependent on the project, the skill sets on the QA team, the timeline, and several other factors.
5. Solve your own problems. The Internet is an incredible tool and offers us all a ton of opportunity to not have to reinvent the wheel. But don't ask other people to do your work for you! Ask for advice. If you want someone to solve your problem, ask one of us to consult (for pay) and bring us in. We'll be glad to get the job done for you!
6. Give back: as you grow and learn, give back... Don't post a question, get your answer, and disappear. Remain an active participant in the community and 'pay it forward'.

Summary? Be specific, research before you ask, solve problems, and give back. That's how to get answers online--in a sustainable fashion.

Technorati Tags: quality assurance,QA,Software Testing

What Makes a Good Automation System (automated QA/Quality Assurance/Software Testing)

So in my new job, one of my first tasks is to put together an automation system--by this I mean a harness and a framework. The process has had me thinking (and talking) a lot about what makes a good system in general.

The automation harness is the system used to schedule, distribute and run tests and to record results. In the open source world, some tools used here include NUnit, JUnit, and TestNG (my personal favorite). These tools all work in a one-off situation - they are all run locally out of the dev environment or via the command line. In software testing at Microsoft, though, a one-by-one approach to automation is useful for 1) developer unit testing, 2) tester unit testing/test creation, and 3) failure investigation. However for the 24/7 test environment we're building, this isn't sufficient. We need a centralized scheduling tool that allows us to push tests out to multiple clients (simulate load and de-serialize test runs). So we're working internally to Microsoft, evaluating the existing automation harnesses available and trying to find the one that works best.

A big factor for me in this selection is finding a harness which is configurable. At Microsoft, we are pushing the envelope in testing in a variety of ways: code coverage analysis, failure analysis, and several similar activities which allow us streamline our testing, reduce test overhead and automate many testing tasks. This means our framework MUST be extensible - we have to be able to plug in new testing activities.

A second element of the automation system is the framework. This is the abstraction layer which separates our automated tests from the application under test. This abstraction layer is critical to good automation. If, for instance, you are automating a web application, you will probably experience a lot of churn in the application layout. You do not want your automated tests hard-coded looking for certain controls in certain locations (ie, in the DHTML structure)--by abstracting this logic, your test can call myPage.LoginButton.Click(), and your abstraction layer can 'translate' this into clicking a button located in a specific div. In some organizations, this framework is purchased. At the LDS Church, we leveraged both Selenium RC and Watij to build this framework, developing most of it ourselves internally (kudos to Brandon Nicholls and Jeremy Stowell for the work they did in this capacity).

The challenge felt by most test organizations is two-fold: 1) finding the engineering talent to build these systems and 2) making the investment in innovation. Ironically, the very thing which can free up resources for other tasks (automated testing) is the thing most managers don't want to spend time on! This makes sense, sort of... managers don't like to invest in activities which (in their mind) don't contribute directly to the bottom line. In all but the smallest of projects, this makes no sense--test automation isn't a sellable product, but if automated tests can free up a day or two of test time, that's a day or two spent doing other activities! Each and every time automation is being run.

Recruiting top talent is also a challenge. In both IT organizations where I worked, there was a culture among developers that testers weren't engineers--they were click testers. Testers couldn't give input on 'extremely technical concepts' like architecture, potential bug causes, or the likes--they're there to pick up code as developers release it, and then to find bugs. It's no wonder that it's so challenging to hire engineers into testing - when they're treated like that, they're either going to leave or move to development!

So the keys to a great automation system are: 1) a solid, extensible and flexible harness, 2) a robust framework, generally customized to your test activities, 3) management commitment to invest in innovation and automation and 4) top engineering talent and the culture to reward them for their contribution.

Am I missing anything?

Technorati Tags: quality assurance,qa,testing,software testing,automation,test automation

What's the Right Tool?

I am pretty active on several testing forums (Microsoft's MSDN forum, and a Yahoo! group for agile testing are the two I frequent the most). I cannot count the number of times someone sent an e-mail asking which big, monolithic tool is the right one for them.

My friend (and manager) has a great response to this. At the LDS Church, we are frequently asked by new testers "When are we going to standardize on an <insert category here - performance, automation, etc.> tool? His response: you're engineers. Look at the tools in use in the organization today, look at the tools available in the industry, and make the best decision for the organization. Sometimes you might give up a little functionality or ease of use in exchange for a tool which is already used in-house. You get a built-in support forum, and sometimes you can even leverage existing tests. Other times, you'll probably pick a best-of-breed tool.

Our performance testing is a perfect example of this. When I came on board, we were using JMeter, an open-source java-based tool. The benefit of JMeter? It replays Apache logs. There's no better mimic of production than replaying production logs! The problem? We have been seeing weird things with JMeter - for instance, if a connection times out, JMeter doesn't drop it and move on. Instead, it opens another connection request but leaves the first request hanging. In some ad-hoc experimentation today, an engineer on my team started with just 5 connections and ended his test at 10 connections. Scale that to the 400, 500, 1000 connections we were using to load up www.lds.org, and you end up with incredibly unrealistic test scenarios! I never felt we could trust JMeter.

So I started looking around, and I have settled on WAPT (Web Application Performance Test Tool, http://www.loadtestingtool.com). It's a commercial product, but only costs $350 per license. There's a lot I love about it - first of all, connection management seems to be so much more reliable. Secondly, the built-in reporting is incredible. Third, there are actual support personnel if there's an issue. Oh, and fourth, I can bring a $250,ooo enterprise server to its knees with a dual-proc Dell running WAPT, whereas I needed 10 machines to do the same with JMeter. I'm pleased as punch with it. However, we've had to abandon a year's worth of test scenarios and rebuild them. I've had to justify my decision to change tools. And I'm the only person in the org running WAPT (right now).

The bottom line is, there's a better way to get advice about tools. Don't ask "Which monolithic tool is the best?". Describe your test environment and the project you're working on, and ask "What are some tools people have used for this type of testing? What are pros/cons/strengths/weaknesses of these tools?" Gather information and make your own decision with advice from others.

You're the engineer on the spot and you need to make the final decision. If you're a test lead in Gurgoan, don't let me decide for you what tool to use.

 

BUT: don't hesitate to ask! If there's one thing most testers have in common, it's an eagerness to share their (opinionated) opinion about something. And I personally love to give advice and help out. This isn't a censor or a rant about asking questions - it's about people needing to step up and be engineers.

 

JTO