Wednesday, October 1, 2008

What A Feeling!

So I've done a relatively good job of being calm and collected in my first two days at Microsoft. Your first day starts in a long line filling in I-9 forms. Then you enter address and other contact info info the Microsoft system via an internal web form. Finally, you sit for another 7 hours in a room while they teach you about benefits and the MS culture. At the start of that first day, you're officially a Microsoft employee, but you don't get your card key or anything.

Day two starts in the Big Room again, with a discussion about corporate ethics, legal issues, and a discussion with a couple of recent Microsoft hires. Finally you get your card key and are sent to find your manager. Oh and by the way - the room has anywhere from 100 to 150 people in it. Yup - Microsoft starts that many people, each and every week (well, maybe not the week of Christmas or New Year).

So at about noon I launched off on my own. Unlike most new hires, having worked here for 11 years, I know where the buildings are, where parking is, etc. So I zipped straight to my temporary building whenever I'm here in Redmond. I swiped my badge and got access to the building. A little smile crept up on my face.

About an hour later, after picking up my laptop and getting it set up, I went for lunch with another new-hire for our Utah group. As I swiped my card and stepped into the cafeteria, I had a completely involuntary reaction: I jumped, throw both hands high in the air, and shouting "Yes!! I'm back at Microsoft!" Later during lunch, Tim told me how cool it was to spend time with someone who is so excited about working at the company. I have a bounce in my step which has been missing for many, many years.

I can't describe it. I was in software testing/quality assurance at Microsoft for 11 years. There were great days and there were really challenging days. I left two years ago to lead QA "for the world's largest retail IT project" at Circuit City. What an experience that project was. Quality assurance to team leads (mostly IBM project managers) meant proving happy path and avoiding negative testing. It was a cultural shock, to say the least. Testing software at the LDS Church was somewhat better. The people, for the most part, were great (surprisingly, there were exceptions - people who behaved less Christ-like than even at Microsoft!). The QA team suffered from a total lack of respect from development, however. And I found in that IT organization that everyone has a special little niche. There are enterprise architects, application architects, security 'specialists' (people who know about security policy, but don't know much about penetration testing), developers, and quality assurance engineers. If you dared to stray outside of your niche, well, that meant you were stepping on someone else's toes.

Back here at Microsoft means I have an equal seat at the table as an engineer. It means I'll have to work with other engineers to tackle really, really challenging problems. First challenge: building an automation harness using existing technologies at Microsoft, then building our own framework (abstraction layer) within which our tests run. Additionally, we need to take the mandate to "build virtualization management technologies" and turn that into a release software application: ideation, product planning, product specification, development, and release testing.

We also have the challenge of hiring incredible C++ developers and testers (engineers) in the Salt Lake Valley. Finding developers is pretty easy, but finding developers who respect QA and understand engineering excellence? A challenge. Finding a software testing professional who has the guts to take an equal seat at the table? A challenge!

But that's what I love about being back. I feel like, after two long years, I can finally do my best work and reach my full potential. I can bring all of my 14 years of engineering experience to bear on a software challenge. If I have an idea, I can run with it. I can provide input to the user scenarios, to application architecture, and to how we push quality upstream. I can prevent software defects, rather than find them!

I know everything won't be perfect. I left Microsoft for reasons, and those reasons haven't all changed (although judging by much of what I heard in New Employee Orientation, the last two years have been a time of growth and improvement for the company). There will still be bad days, there'll still be the struggle for a good work/life balance (now called 'blend'). But I have incredible health benefits for my family, stocks and bonuses again, and I have the chance to be challenged every day again. And I'll be working with super-smart people every single day. THAT is a cool thing!

Monday, September 22, 2008

A Softie Again

I'm thrilled to announce that I'm about to become a Microsoftie, again! I have been given the opportunity to return to Microsoft as a senior test lead, working in the Management Division. No, that's NOT the group of execs who run the company! That's the group producing OS and platform management tools like Operations Manager (a product I've worked on in the past), SMS Server and so forth.

About two months ago, Microsoft announced the formation of a development center here in my home of Salt Lake City. I quickly applied for roles in the test organization. After two years of working project IT, I am ready to get back to product software! Some people might snicker, but the commitment to quality at a product software company, especially Microsoft, is so much higher than in project IT. I found I was spitting in the wind quite often in my previous roles--I was either fighting losing battles or I was fighting the wrong battles (ie, pushing for quality in organizations or situations where leadership didn't share the same commitment, or pushing for higher quality than some projects required).

Additionally, I found that my last few organizations did NOT look at testing engineers as equal citizens. Testing was looked at primary as QC (quality control) or possibly QA (quality assurance). Rarely were we invited to planning or design meetings, never were we looked upon as people who could help PREVENT defects. We were often seen as a roadblock to release. Don't even ask about my two-hour argument with IBM about the value of negative testing! Engineers who could help architect solutions? No way!! I often felt my fourteen years of engineering experience were swept under the table because I had "QA" in front of my "Engineer" title, rather than "Java" or "Development".

So I'm very excited to be returning to an organization and a company where I get a 'full seat' at the table, and an opportunity to contribute to my full ability, not to the perceived level of my title. IE, I'm an engineer again!

Doesn't change much about my point of view or my approach to testing. I'm still hyper-focused on 1) bug prevention, 2) quality engineering, and 3) driving for greater effectiveness and efficiency. And sometimes that will put me at odds with mainstream MS thinking (recall my conversation about moving to all SDETs and automating all tests, and the resulting "user experience" bugs that slip through the cracks in products like Vista).

For me the best thing is, I get to stay in beautiful Salt Lake City, and yet work for one of the greatest employers in the world. And NOW I can honestly say I've tried a few others...

So OK - let's hear it. How bad is Microsoft quality, really? If you could talk to a test leader, what would you tell them? What would you ask them?

Friday, July 25, 2008

Software Quality Assurance: SDL (Secure Development Lifecycle

At work, I've been helping one of my teams implement portions of Microsoft's Secure Development Lifecycle (SDL). SDL is more than just plinking around attempting some penetration testing--it's a committed approach to secure software design. Here are some of the takeaways from our work:

  1. The first point I made with my team is that security features DO NOT EQUAL secure features. Having SSL encrypted communications does not make a web application secure! It just means you have an encrypted communications channel. Secure software isn't secure because of features such as Acegi security, RSA encryption or anything like it. Secure software is produced when developers think secure from the start. Secure software comes when code is written safely, when developers write solid, secure code, and when testers are helping with the planning and testing of the software with a secure focus.
  2. Next point we emphasized: threat modeling. As we wrapped up a two-hour threat modeling session, one of the developers commented "Why didn't we do this months ago, before our code was written!". Good point!! It's never too early to threat model. Analyze your product, paying special attention to where data crosses boundaries: user to Internet, Internet to server, server to database, etc. Model threats, wild and crazy or down-to-earth. Our threat modeling has resulted in 9 potential threats so far, and we expect several more as we continue.
  3. Security comes in layers. Back when I lived in India, I toured the Delhi fort. This fort was built by professionals. It has a deep moat around it. Tall, thick walls surround an inner wall, and inside that inner wall lies the fort. That's how our code should be! OK - so you're authenticated via Acegi and LDAP. You're encrypted with SSL. That's great - but what if someone logs in with a valid account, then tries to hijack another session? Your layered security will catch hacks like this--authorize anytime someone tries to access sensitive data. Even if you've already authenticated and authorized, do it again! Layers bring security.
  4. Reduce your footprint: in Agra, where the Taj Majal is, there's another fort (these Moguls were building forts everywhere!). This fort is on the edge of town, in the hills. Compared to the Taj, the fort is tiny. This is referred to as attack surface reduction. Only allow public access to a few of your resources. If you have features which suffer from weak security, disable them by default or remove them completely. Give hackers as little space as you can.
  5. Train your engineers (dev and test). There's common training needed by both (elements of secure design, running a threat model, etc.) and there are discipline-specific trainings such as penetration testing or the application of specific technologies. The SDL is called a lifecycle because it's a continuous process. Lather, rinse, repeat and all that.

Our training has produced benefits. For starters, developers have a new security-focused mind set. We've found a few security bugs already, and our threat model has exposed some potential issues. This is great progress, and it comes from just one day of work. Imagine what we'll be like in a few months after a day or two of training and a complete milestone with security in mind!

It's never too early or too late to take a step back and start thinking security. I designed our course based on my experience at Microsoft, which was neatly documented in Michael Howard's new book "SDL: Secure Development Lifecycle". I cannot recommend reading this book enough!

Got a security question? Post it here...

Tuesday, July 15, 2008

Software Testing Patterns: Sessions

One of the most common things to have to test is sessions, cookies, and login/logout. This is the start of a pattern for that kind of testing.




First, some background:




Test Patterns:





  • Session hijacking: create several cookies for your site (varying login sessions). Tests: 1) can you substitute the session id from one session and use it in another session. 2) what happens with invalid session IDs? 3) Prevention: specify that cookies must be https-based (specify the secure flag when creating the cookie.


  • Cookie poisoning: change values in a cookie - for instance, if a cookie stores the check-out value in a shopping cart, the user could change that value before checking out.


  • Back button: do something that adds/modifies a cookie, then click 'back'. The state could then be out of sync (being on a page which expects you to NOT have a value in your cookie which you actually do have).


  • Expiration: cookies w/o expirations are considered 'not persistent' and should be destroyed at the end of the session. Persistent cookies represent a security risk due to how long they are persistent; make sure the expiration of a cookie is reasonable (30 min?) based on typical user scenarios.

  • Log in, thereby picking up a sessionID. Make note of the sessionid. Close the browser, and then log in as someone else. Finally, update the cookie to be the id you noted previously, and hit refresh. Does the session change?

Software Testing Patterns

I'm convinced there are a series of testing patterns we could use. I haven't found any on the Internet, which is surprising, but I figure I'll start documenting them here. Look for blogs with "Test Pattern" in the title.

Don't be shy about contributing, please!

First of all, some links:

Quality Assurance or Testing Seminars In India

I've often wondered if there's a market for QA/quality assurance/testing seminars in India. I've thought of setting up seminars for how to test in cities like Mumbai, Bangalore, Delhi, Chennai. One day seminars where people pay a flat fee to get in and we spend the day really digging into testing. Some presentations, some 'labs' and a lot of Q/A.

If you think there's value, post a comment. If you'd be interested in partnering, hit me in an e-mail joatgm at gmail dot com.

Should Freshers be Testing Software?

I have seen an alarming trend in India and worldwide. "Test engineers sought - freshers" (for those not familiar with the Indian IT scene, a 'fresher' is someone fresh out of college - 0 to 1 year experience, maybe even a little more).

Seems like everyone with an open tester position wants to throw a fresher at it. This is a huge mistake! What does a fresher bring to the table?
  • Enthusiasm
  • Lots of book learning; in India, that means rote memorization
  • Some programming experience.
I once helped a large, failing consumer electronics company move work offshore to save money (offshoring to save money is a BAAAADDDD idea--ask me why or wait for a blog on it). The development 'partner' on the project brought in a ton of freshers and some somewhat seasoned testers. What did we get? We had test engineers who didn't know what a defect report was, who'd never used a defect tracking tool, who had absolutely no domain knowledge. All they had to contribute was 1) a warm body and 2) some experience in Java. This is a recipe for disaster.

Is there room for freshers in an IT organization? Absolutely! Microsoft focuses most of its recruiting in the college space, but that's because the ratio of new hire to full timer (at least in the US) is probably 10:1. They have people all around them who can mentor and grow them, and they are NOT encouraged/forced to do mundane and boring tasks. Well, not all the time... As a new hire at Microsoft, I was the release lead for German Mac Office 98, with 18 months experience in the company! But that was after 18 months of incredible mentoring and learning.

For you people hiring for testers, don't think "Well, no one wants to be a tester - let's find someone who wants a foot in the door and get them in here." You're doing yourself a disservice, you're doing your customer/project a disservice, and you're really not helping the fresher either. Look for some passionate QA/testing professionals. Bring them in and let them establish a world-class test organization. Then hire freshers, train them up, and keep growing your organization.

OK - off my soapbox for now...