How Can I Become a Better Tester? Part IV: Mentors

Technorati Tags: testing,qa,quality assurance,software,engineering

A major step for career growth, in any role, is to find a mentor. Mentoring relationships come in many forms, the most common of which are:

• Working for someone
• Working with someone
• Having a formal mentoring relationship
• Having an information mentoring relationship
• Reading and participating in specific test focus groups

Working With Someone

The best way to learn from someone is to work with them, elbow to elbow. There's no better way to learn. I think of a couple of great test managers I had when I was a test lead. Dan, a test manager in Microsoft's Mobile Devices Division, is a fantastic manager. He is great with people, protects his teams from politics and struggles above, and understands quality. One thing I learned from Dan is that, while a tester's appetite for more time or resources is never satiated, we can get the job done anyhow. When I complained to him that we shipped a product with too few people, he asked "Did it ship on time?" [Yes] "Have there been any recall class bugs?" [No] "Then you must have had the right number of people."

Another great mentor was Mike, group test manager for Live Meeting. Mike and I didn't see eye to eye on everything, but what a great manager he was! He enabled and trusted people to do way more than they may have ever done. He put me in charge of a beta release of Live Meeting (then called Placeware), and let me drive shiproom meetings of several releases of Live Meeting. He also knew how to have a lot of fun on the job. I miss many aspects of working with Mike, frankly. And I try to make every team member's experience the same - lots of opportunity to grow, lots of fun, and high expectations.

I'll make a wild statement: in your first few years of your career (generally two to three) WHO you work for matters significantly more than HOW MUCH you make. The first years of your career establish a foundation which will determine how quickly you will grow and what kind of habits you will form. If you're fresh out of college and have a choice between working at lower pay for an incredible lead, or earning more and working at a 'code factory', may I recommend you take the former? Establish yourself early on in your career, learn the principles, and THEN go out where you can impact and be rewarded accordingly.

Working With Someone

Almost as good as working for a great test lead is working with a great tester. At any level. In Microsoft as a junior tester, I worked side-by-side with a person who became a great friend. He taught me about equivalence classes, boundaries, and other key concepts. He showed me how to fight for bug fixes. He showed me where the bar should be!

Twelve years later, I was managing a team of 100. I worked with three test leads who taught me a lot. Sri taught me about getting the job done - he just sticks to the job until it's done (I've always prided myself in being known as a person who gets the job done, but Sri showed me how to take this to the next level). Debbie taught me about putting your head down and working through challenges - stick-to-it-ness, if you will. And Jenn taught me about taking pride in whatever you do. We never stop learning, especially by example.

My current manager is much the same. I have never worked for anyone as diplomatic as Tony (and I don't mean that in an office-diplomacy, fake-smile way). He really cares about people, how they feel, and what they think about their job. He's also ready at any moment to take advantage of a teaching/mentoring opportunity.

Having a Formal Mentoring Relationship

At Microsoft, each new hire had a new-hire mentor for their first three to six months. This mentor was the go-to for pretty much anything, from "where's the printer?" to "do you think I'm making my goals?" After that, everyone was advised to find a mentor within the company, and Microsoft even has an internal site dedicated to finding and maintaining a networking relationship.

The same should be true for you. If you're new to a company, I recommend you find yourself an internal person to mentor you. Once you're established, look around and find yourself a mentor. Generally you want to find someone who's ahead of you in some way (technical, project management, leadership skills, etc.). Ask them to be a mentor and be very protective of the time you take from them - make sure every time you meet you have a productive conversation. Never ask them to do your work - ask for help reviewing what you might think is the right proposal, and get feedback into specifics.

Have an Informal Mentoring Relationship

A key requirement for me is to work with great people from whom I can learn. As I pointed out, I learn from people I manage. I try to learn in almost every circumstance and every person. If there's someone you learn from a lot, try to be around them a lot even if you're not in a formal mentoring relationship.

Participate in Forums, Groups, Discussions and Seminars

Finally, there's a lot to be learned by reading and participating in forums, groups, discussions and seminars. I'm active on several forums (MSDNs testing discussion forum, agile testing forum in Yahoo Groups, etc.) I learn a lot just by lurking (although I'm so opinionated that it's impossible for me to lurk for long) or by joining in on the conversation. Other places to learn include seminars, networking groups, and even podcasts (BTW: stay tuned for a podcast from me over on http://www.searchsoftwarequality.com, where I'm a Testing Expert)

How Can I Become a Better Tester? Part III: Going Beyond Stated Requirements

So you've spent some tme becoming more aware of quality and what quality means. You've been looking at the differences between a Mercedes and a Hyundai. You've also started reading up on quality and on engineering. Great start! What's next? Well, the next step is to realize you need to look beyond the stated requirements and dig deeper.

In my opinion, functional or business requirements docs are like nets. They catch a lot of 'big stuff' but they can easily let little--albeit important--stuff through. For instance, a business requirement might be that a tax calculation be performed on a per-item basis. t the same time, it might not mention that the overall tax rate, which is rounded to the lowest cent, needs to be calculated on the total purchase and not the individual items.

Disclaimer

I have no idea what the rules are for tax calculation. I don't test it, never have. And even if I did know, it'd only be specific for the US. So please - look over the details in this example and try to recognize the key point...

End Disclaimer

The Tax Man Cometh

As an example, assume a user buys one $0.55 candy bar, a $1.75 bag of chips, and a $4.27 bottle of antacid. At 10% tax, that would look something like this:

Item	Cost	Tax	Total
Candy bar	$0.55	$0.055	$0.60
Chips	$1.75	$0.175	$1.92
Antacid	$4.27	$0.427	$4.69
			7.21

 

Tax is calculated on each item, and in all cases is rounded down to the lower cent.

However, if you factor tax on the sub-total of the items, it may actually be more! The subtotal of these three items is $6.57, and tax is $0.65. Calculated this way, the total is actually $7.22.

OK - I agree. This is a case of a missed business requirement. But this is a great example of how a tester needs to be on his or her toes, asking questions and always exploring for what can go wrong.

Not Another Bad Lego!

I'll take a manufacturing flaw as another example, which hopefully you can extrapolate to software quality. I have three sons, spanning just 4 years, and as a family (Mom is the ringleader here), we are Lego entusiasts. Birthday presents are almost always Legos - these days, they are super-complex Star Wars sets. Whenever my boys have saved up $10 or so, we're off to the store to buy a new Lego. Several times in the past couple months, we have picked up Legos with one of two problems: 1) missing peices and 2) poor fit. There's nothing more frustrating for  a kid than to have their new Lego missing a critical piece--or for the parent who has to drive 20 miles to return the Lego!

I'm sure there's a manufacturing QA requirement here that each individual bag of pieces has a certain total weight. That's how they make sure all the pieces get into the box (how they miss a piece on this test is beyond me). Two days ago, my youngest bought himself a new Bionicle, only to find that the Bionicle's shoulder-claw (you have to see one to understand) was in the box but it didn't stay in place. Turns out, a manufacturing flaw caused the shoulder claw's insert tab to be too small, producing insufficient friction for it to stay.

So as a QA engineer at Lego, I'm sure I'd be dealing with the requirement to have all the parts in the box. But would I be asking about how we ensure EACH part shipped in EACH package has the right fit? That's going beyond the requirements.

Hey, You Stole My Car (Analogy)

OK, one last analogy and I think the point will be well-clarified. Let's pretend you are responsible for quality for an automobile. The requirements are that it be capable of a certain miles per gallon (KMPL), that it run for so many successive hours, and that it fit so many passengers. So let's assume these are 25 MPG, a service life of 100,000 operating hours, and it seats 4.

In the early 80's, AMC made a car called the Pacer. It was small, rather fuel efficient, and cheap. It sat five (although we had many more than that in my friend's Pacer once). Service hours? My friend Kate couldn't kill it, no matter how hard she tried - she drove it for a year without putting oil in it!

A colleague of mine at Microsoft recently had to buy a car. He chose a VW Toureg V10 diesel. He actually hits 30, 35 MPG. The car has a proven service life well in excess of 100,000 hours and it fits 5 (very comfortably, I might add). Which car is of higher quality? Why?

I'll leave you with that thought. Dig deeper, go beyond the stated requirements. It's the difference between an Ambassador and a Skoda, a Pacer and a Toureg.

How Can I Become a Better Tester? Part II: Read

OK - so you are paying more attention to quality now, right? Starting to ask more about what makes a Mercedes so much better than, say, an Opel or a Dodge. What's another step you can take to become a better tester? Simple - expand your horizons by reading. That's what this blog entry is about: what can you read?

• The first place to start comes for free: read off the Internet. Read blogs (http://www.testingreflections.com is a great one--there are many out there). Also, a site I'm affiliated with (http://www.searchsoftwarequality.com) is a great resource. There is a performance test expert who frequently contributes, as well as several experienced test managers.
• Related to blogs, start reading test forums. I have found that the Yahoo Agile Testing Group is a great source of information. Another good source is MSDN's forum on software testing.
• The next place is to read books on testing. Cem Kaner, James Bach, James Whittaker are all great authors with significant experience. Whittaker has a series of "How to Break..." books which is rock-solid with good day-to-day steps for your testing.
• The next great source of reading is books on the topic of engineering. These books talk about best-practices in teh engineering discipline and always lead me to think about how my team is approaching a project or a problem. Some of my best test strategies have come as I've read development manuals such as:
• Test Driven Development with Microsoft .NET (Newkirk and Vorontsov)
• The Object-Oriented Thought Process (Weisfeld)
• Software Project Survival Guide (McConnell)
• Writing Secure Code (Howard)
• Conferences and seminars: let's face it, a lot of conferences are actually just about marketing something. But even those conferences offer the opportunity to sit with experienced testers and learn from them - either in presentations or during the 'chit-chat' portion of the conventions. A couple of thousand dollars is a lot of money to spend on a conference, but sometimes it'll pay back in spades, just with the new network of associates you build who can help you through a challenge you may be facing. I'm speaking in May 2008 at PSQT's Vegas conference, if you're interesting in meeting me face-to-face.
• Certifications: OK - to be clear, certification is NOT a substitute for experience. It alone means nothing, but a certification can often lead you in the right direction in improving your skills. It can build a framework for you to 'flesh out' with experience over time. PSQT offers certification and there are a few other courses available. Again, I stress: certification in and of itself can only teach you the process and steps recognized as historically good testing. It's not everything you need to be a great tester, but it is a good framework.
• Practice: the best teacher is practice, and that's what you really need. Be really weird - start a testing group which meets after hours and just tackles weird testing problems (meet at a local cafe and talk about how you'd test the espresso machine or the retail point of sale system).

These are just a few ideas. There are many, many ways to learn. The key to learning, however, is wanting to learn - and for the right reasons. If you want to be a better tester so you can move up the food chain and become a better manager, well, you'd better think about a career as a project manager instead. If you want to be a better tester and that's it, trust me - you'll learn. And the growth opportunities will present themselves, too.

Next subject: going beyond the requirements.

How Can I Become a Better Tester?

In all my blogging and participation, I have met many testers. One recently asked me "How can I become a better tester?" As I answered, I thought "Hmm - might be a good blog series!" So here we go. Most of this is aimed at beginning testers (1-3 years) although I have to admit I would benefit from some touch-up work in each of these areas!

I gave my friend four steps to take. I'm sure I'll come up with more, but let's look at these four briefly, and then I'll spend a blog or two on each as time goes by (hopefully I'll finish the last one sometime before the Rose Bowl).

1. Become aware of quality - what is it, what does it look like, what does it feel like?
2. Read - read everything you can get your hands out about quality. Start off with books about testing. I really don't even care about what testing methodology (agile, monolithic beastly projects, etc.), just read.
3. Related somewhat to (1), recognize that quality means going beyond functional and business requirements. You need to think of yourself as the gateway to quality or, in an agile world, as the trustee for quality. Teach people about quality and manage up for quality.
4. Find a great test lead, test manager, or test mentor to work with. If you can be employed on the same team, that's great. If you have to be in a less formal mentoring relationship, that's great. Learn from this person, everything you can.

First: become more conscious about quality. I actually gave my friend a homework assignment - as this person lives in India, I asked them to compare two vastly different automobiles: the Ambassador and the Skoda. Ironically, the Skoda is the butt of many a joke in Eastern Europe, but it's actually been taken over by VW/Audi and, internationally, is an incredible automobile. It competes with (and beats) Honda in India as the ultimate status car within reach of the rising middle class. The Ambassador? Well, it's truly the Indianized car--it's one of two cars mass-produced throughout India's history. When you compare the quality of the two, well, it's like comparing a Chevette to a Lexus in the US. Or, I suppose, a Trabbi to a Mercedes in Europe. No offense intended towards the Ambassador--it's just reality.

So I like to think about quality and compare products and services. If you live in the US, try this - go to Mervyns or Kohls to buy a suit. Then go to Nordstrom. The difference is beyond belief! If you don't think consciously about it, though, it's hard to put into words what the difference is. A good business analysis would miss the point either - consider the business requirements:

• Stock on hand: variety of sizes, multiples of the most common sizes. Mervyns: check. Nordstrom: check.
• Merchandise is on the rack, readily visible to the customer: Mervyns: check. Nordstrom: check.
• Sales reps available to answer questions: Mervyns: check (you do have to seek them out). Nordstrom: check.
• Someone who can ring up the transaction: Mervyns: check. Nordstrom: check.
• Merchandise is priced right: Mervyns: check. Nordstrom: check.

A tester who focuses on only the business requirements here would say both companies are of equal quality. But think deeper - look at the experience in both situations. The Nordstrom experience is full of stuff that could be measured, but it's necessarily the first thing you think of. How friendly and helpful is the service? What is the atmosphere like? What's the merchandise like? Will it feel cool during summer and warm in winter? Is it scratchy? Is it comfortable overall?

So the purpose of the homework is to think deeper about quality. Is quality just meeting the requirements, or is there more? Is it about understanding the customer and knowing there are often unwritten requirements?

How would this apply in the real world? Well, I've already blogged about a huge data corruption issue that arose at a previous employer because the consulting firm running an implementation there insisted negative testing wasn't necessary. They focused (barely) on proving the software met requirements--since none of the requirements discussed issues like "the data needs to have integrity" or "the client must respond to and handle error messages from the server", no testing was done in that area. Thirteen million corrupt rows later, four workstreams came to a halt while the data team narrowed the issue and implemented a fix - oh, and fixed the corrupt rows.

Becoming aware of quality is the first step to contributing to quality. It's what helps us lift our eyes beyond requirements and to be stewards of quality.

Technorati Tags: QA,Quality Assurance,Software Testing,Testing,Quality

What's the Right Tool?

I am pretty active on several testing forums (Microsoft's MSDN forum, and a Yahoo! group for agile testing are the two I frequent the most). I cannot count the number of times someone sent an e-mail asking which big, monolithic tool is the right one for them.

My friend (and manager) has a great response to this. At the LDS Church, we are frequently asked by new testers "When are we going to standardize on an <insert category here - performance, automation, etc.> tool? His response: you're engineers. Look at the tools in use in the organization today, look at the tools available in the industry, and make the best decision for the organization. Sometimes you might give up a little functionality or ease of use in exchange for a tool which is already used in-house. You get a built-in support forum, and sometimes you can even leverage existing tests. Other times, you'll probably pick a best-of-breed tool.

Our performance testing is a perfect example of this. When I came on board, we were using JMeter, an open-source java-based tool. The benefit of JMeter? It replays Apache logs. There's no better mimic of production than replaying production logs! The problem? We have been seeing weird things with JMeter - for instance, if a connection times out, JMeter doesn't drop it and move on. Instead, it opens another connection request but leaves the first request hanging. In some ad-hoc experimentation today, an engineer on my team started with just 5 connections and ended his test at 10 connections. Scale that to the 400, 500, 1000 connections we were using to load up www.lds.org, and you end up with incredibly unrealistic test scenarios! I never felt we could trust JMeter.

So I started looking around, and I have settled on WAPT (Web Application Performance Test Tool, http://www.loadtestingtool.com). It's a commercial product, but only costs $350 per license. There's a lot I love about it - first of all, connection management seems to be so much more reliable. Secondly, the built-in reporting is incredible. Third, there are actual support personnel if there's an issue. Oh, and fourth, I can bring a $250,ooo enterprise server to its knees with a dual-proc Dell running WAPT, whereas I needed 10 machines to do the same with JMeter. I'm pleased as punch with it. However, we've had to abandon a year's worth of test scenarios and rebuild them. I've had to justify my decision to change tools. And I'm the only person in the org running WAPT (right now).

The bottom line is, there's a better way to get advice about tools. Don't ask "Which monolithic tool is the best?". Describe your test environment and the project you're working on, and ask "What are some tools people have used for this type of testing? What are pros/cons/strengths/weaknesses of these tools?" Gather information and make your own decision with advice from others.

You're the engineer on the spot and you need to make the final decision. If you're a test lead in Gurgoan, don't let me decide for you what tool to use.

 

BUT: don't hesitate to ask! If there's one thing most testers have in common, it's an eagerness to share their (opinionated) opinion about something. And I personally love to give advice and help out. This isn't a censor or a rant about asking questions - it's about people needing to step up and be engineers.

 

JTO

WWII and Test Management

I've been on a WWII kick again, reading a lot about the war. Eisenhower's job was to decide on the strategy - do we attack Germany in one concentrated punch (Montgomery's plan) or do we attack along the entire front (Eisenhower, Patton's plan). Patton's job was to figure out how to implement--he had to sit and watch as Montgomery's punch plan failed in places like Arnhem. Then he got to decide strategy for crossing the Rhine and driving for the win - Berlin or the Eagle's Lair? Once these decisions were reached, they were carried down the chain of command - there was a big marshmallow layer of bureaucrats who procured supplies, managed replacements, etc. and then - and then there were the lieutenants.

The senior leaders (generals and majors and such) are the test managers. High-level strategy, supply mangement, personnel. They're negotiating at a high level with PM or dev on things like engineering process and such. They may be fighting over budget and headcount, too. The lieutenants? These were the boots-on-the-ground leaders who took initiative and got things done. They trained for D-Day for over two years, learning to climb cliffs and take out bunkers. The day after D-Day, they encountered hedgerows and had to make up a new strategy on the fly - and good thing, too, that they did it--there were there, in the heat of battle, and they knew what worked and didn't. They called in support from the Navy or Air Force. These are your test leads, the folks rallying the troops. They're the folks who should decide to implement MBT or PICT or the likes. They're the ones who need a great working relationship with dev leads (something I'm not doing great at right now myself - need to work on that). They're making sure the privates are moving along, they're training new leaders, and they're calling for sacrifice at appropriate times.

Black Box Testing - time to go?

Regarding black box testers... Having lived in both product software AND IT engineering worlds, I'm beginning to understand the root cause of the argument about the value of blackbox testing. Developing user product software requires a ton of testing; at MS, we did this against more functional specification rather than business requirements. I'm convinced software applications (even apps like Oracle Retail, which suffer significantly from the lack of blackbox testing) need serious blackbox testing because the target audience and the way the product will be used are so varied. Integrating business applications in an IT setting, however, really doesn't need as much and yes - I believe this could be covered by business personnel rather than by QA engineers.

But here's the conundrum... Until the IT industry accepts testers as engineers and allows them to work with dev as code is developed (and this on a day-to-day norm rather than an exception to the rule), it'll be difficult to convince engineers to go into testing and to stay there. In my role as TM for Circuit City's MST project, I interviewed 30-40 people for open roles and found practically no one who could code. None of my 30-member team in Richmond could code beyond writing XML (well, one could--he was my best test lead, too). Very few of the personnel provided by IBM, the engagement partner, could code either. One tester they provided, who had a rather high bill rate, was justified b/c she had 'retail experience'. She worked in a clothing store in NY during college!

So there's a chicken/egg thing which has to be solved one company at a time. Convincing management to 'up the bar' for test engineers, convincing development to be more agile, etc. are all necessary steps and I think it's still going to take a while. I for one am totally convinced that you get more for your buck with whitebox and gray box testing, and that you get way more for your buck hiring technical testers. But convincing a CS grad that she should consider test when all the glamour is associated with development? That's a challenge.

All that having been said, don't scoff blackbox testing. A great test engineer (very rare commodity) is a good programmer who excels at white box testing, who can black box test as well. Those finishing touches, esp in product development, are critical to project success.