Blitz Blog: Quality Assurance Through Code Analysis

Today I read through a Parasoft whitepaper published on searchsoftwarequality.com, and I found it to be a great approach to static code analysis.

FULL DISCLOSURE: I write articles and am a “Software Testing Expert” for searchsoftwarequality.com. However, I do not blog positively if I don’t believe in the posting.

Parasoft is the manufacturer of an application security static code analysis tool. Of course, the white paper’s intention is to sell copies of their tool. Can’t fault them for 1) believing in their product and 2) wanting to pay the bills.

What I really like about this article is the approach they take to SCA. They are not pushing it as the end-all, be-all of code quality. They are very realistic about SCA, in fact, stating that it’s often over-used and, once over-used, ignored. Companies implementing SCA must be careful about it—don’t enable a rule unless you really want to enforce it.

At the same time, they make a strong point about the value of SCA. It can really help a team drive quality upstream. Some policies an engineering team might want to use, for example, contribute to code readability while other contribute to security (dynamically-built SQL statements vs. parameterized queries anyone?).

As an Agile engineer, I do take issue with their heavy-handed ‘management enforcement’ method. Agile teams need to adopt policies as needed. Cross-company Agile team representatives might establish company-wide policies and enforcement, but the Agile team itself should arrive at the bulk of the policy definitions.

One thing I like to see is the implementation of SCA directly in the build process – ie, no build if the code fails with errors, and a team-wide email on warnings. This is the best way to enforce policies (but teams need to be selective about policies, so they don’t overwhelm or ‘over-stay their welcome’).

Anyhow, blitz blog. Highly recommended reading!

http://go.techtarget.com/r/6673725/7930283/1

Technorati Tags: quality assurance,software testing,qa,security

Oracle Owns the Stack

You’ve probably already heard that Oracle has acquired Sun Microsystems. It’s one big, happy, expensive enterprise systems family! Personally, I think this is not a good thing—it hampers competition. Oracle and Java already have a stranglehold on the IT space, and this merger just tightens the binds.

I have worked a total of 12 years at Microsoft, with two years in IT at other companies, which were both very big Oracle companies. I worked at Circuit City in the Oracle Retail implementation, and I worked at the LDS Church where most everything is running in Java on AIX. With my two years of experience, after working in Microsoft technologies, I have decided Oracle is overpriced, overhyped software. Java I could take or leave, but I firmly believe Oracle sells too little for the money.

Here’s the deal… Oracle sells their hardware and their software at astronomical prices. I mean, seriously… The hardware is expensive. The OS can often be open source, but it still requires people to manage it. If you add up the hardware, DB and other proprietary software, and the management software you’ve sunk a ton more money into than if you’d bought from the ‘Evil Empire’.

On top of that, I do not believe Oracle sells decent products. SQL is OK, but Oracle Retail? It’s a mish-mash of third-party acquisitions, none of which really share a common language or data format. And they bundle it all together and say “It’s Oracle – it’s enterprise quality” and people believe it. As a QA manager, I was appalled. For instance, I could NOT get the Oracle account rep on-site for Circuit to tell me what level of security testing the product had undergone. He told me that was proprietary information which he could not share. So we were investing millions in his software, and we didn’t have the right to know if it was secure? Another example: we tried for weeks (I paid three IBM consultants nearly $600 an hour for six weeks) to automate one of Oracle Retail’s components. The entire time, the Oracle rep was in our status meetings listening to us report blocked, unsuccessful, struggling… Never once did he peep up, until I finally asked how they automate regression testing. At first, he tried to evade with the same “that’s proprietary information” response. When he finally realized I wasn’t going to drop the issue, he admitted they didn’t automate anything in that area. What?? What kind of company watches a customer throw thousands (tens of thousands) of dollars down the drain, all the while knowing the effort is futile?

In my role as QA Engineer at the LDS Church, I was surprised to hear people tell me they would not deploy Microsoft products in the public-facing network segment because Microsoft controls the entire stack, and a security exploit in one layer of the stack meant the entire stack was at risk. ??? From a technology point of view, the opinion makes no sense. Ironically, these were the same people who approved of and supported internal development of security technology, when Microsoft had an existing product on the market (and deployed at the LDS Church). With Oracle now owning Sun, will that answer have to change?

So in a recent guest posting on HISTalk, Orland Portale (former GM, Global Health Industry) spouts on and on about how good this acquisition is. To me the nail in the coffin is the statement that Oracle will begin creating “tightly bundled system stacks which incorporate hardware and software components. Oracle will now have all layers of the systems stack under its umbrella, including the storage, server, operating system, programming language, database, Web services, etc.”

In recent years, Oracle has been snapping up enterprise solutions left and right. They own the entire retail suite. They bought BEA, one of the last independent web providers. They recently purchased MySQL and have control of it. At what point will all the “Evil Empire” opponents realize there’s a newer, much larger empire out there – and it controls all the software they deployed because they wanted to stay away from Microsoft technologies?

I know… Long rant about this. I admit I’m a Microsoft bigot. Except for one product (SMS, or System Center Cofiguration Manager), every enterprise server product I have deployed has just, well, worked! And been reliable and stable. So my bigotry is based on my personal experience. It doesn’t take a PhD to deploy MS software, it works well together, and it’s got great uptime. And the price is approachable, too.

Soon I think it’ll become obvious: Microsoft is the underdog here. Fight back, stop the Evil Empire. Buy Microsoft!

Tenants of a Software Testing Team

If you had the chance to write out the tenants (unbreakable ground rules) by which your quality assurance team lives, what would they be? Here are some I’ve been thinking of:

• We partner actively in quality-first development, by participating in the entire SDLC and by contributing tools, process, and time to the development activity.
• We do not and cannot ‘test in quality’. We test to validate requirements, discover missing or inaccurate requirements and implementation, and to expose defects.
• We use tools and process to discover defects, validate functionality, and improve quality. Not to use tools and process.
• Secure software is a top priority—we protect our users’ privacy as well as our applications reliability.

What about you? Do you have other tenants on your team?

Technorati Tags: quality assurance,qa,software testing,engineering,leadership

Blitz Blog: John Glaser on the Healthy IT Org

Departing from my quality-focused blog postings for a while (software testing rules, but hey – sometimes there’s good info elsewhere too), here’s a pointer to something you need to read!

John Glazer is a CIO and contributor to HIS talk (THE blog for healthcare IT). He captures the essence of a healthy IT org so well, it’s a must-read. And it is quick!

http://histalk2.com/2009/04/13/being-john-glaser-41409/

Technorati Tags: quality assurance,qa,testing,software testing

Blitz Blog: Persistent Quality Assurance

This is a blitz blog – quick, to the point, and hopefully helpful. Topic: persistence.

My wife has a cat (a HUGE cat, although my aspiring veterinarian son tells us she’s a ragamuffin breed, which is ‘big boned’ by nature). This cat I do not like. Don’t know why, but we’ve never really gotten along. But lately, she’s really been trying. She comes into my office, right next to my desk, lays down, rolls over, and just purrs. All the while, she looks right at me with the kindest, happiest look on her face. Finally, I give up and pet her.

Can we learn from the big cat? When we want something, can we be pleasantly persistent, just waiting patiently for the attention we’re looking for? I can look back on my career and see many times where some pleasant persistence might have paid off better than being a bit more forceful).

I’ve been reading “Fearless Change” and have found the message in this book to be the same persistent patience that our cat is demonstrating.

Technorati Tags: quality assurance,qa,testing,software testing,engineering

Software Testing: Is Your Quality Assurance group a Thermostat, or a Thermometer?

I frequently read a daily thought on families and how to improve your family. Yesterday’s thought dovetailed neatly into something thinking I had on QA organizations: are you a thermostat, or a thermometer?

A thermostat is a device which regulates the temperature in your home. It controls the atmosphere. A thermometer is a device which reflects the temperature. To be a thermometer is to be reactive; all it does is indicate the temperature. To be a thermostat means to proactively set the temperature.

In the family, a ‘thermostat’ parent doesn’t react to grumpiness or other negative emotions, but calmly sets the tone in spite of what’s going on around her or him. A ‘thermometer’ parent, on the other hand, gets ‘set off’ easily when children bicker, whine, or complain. Obviously my goal as a husband and father is to be a thermostat, although there are ‘thermometer-like’ days!

In a recent thread on the Yahoo Agile Testing Group (about continuous release), I noticed more than one poster saying something like “I tell management about an issue, and let them decide.”  This reactive approach to QA has been a pet-peeve of mine. It smacks of cop-out to me, and I’ve seen it develop into a victim mentality among many testers—myself included.

In my tenure at Microsoft (where I’m ‘serving’ my second stint, with a combined tenure exceeding 12 years), the testing organization has been consistently active in pushing for quality. A few things differentiate Microsoft from other organizations: first, the engineers wear the pants (that’s changing, slowly). In fact, nothing ships if the program manager, development manager, and test manager sign off on it. And they won’t sign off unless their teams sign off. Secondly, any individual has the right to stop a project, even the newest tester in an organization. Third, while the management structure is massively deep (I’m some six or seven layers below Steve Ballmer, current CEO and President), any individual can get the ear of their VP, Senior VP, and CEO.

So the culture I’m most familiar with is one wherein testing is an active participant in product development. The testing organization moves beyond a ‘show and tell’ relationship with management to an active participant in business strategy. The testing organization is part of thermostat, and rarely plays a thermometer role.

I have worked in organizations where this is not the case, and I’ve seen a lot of evidence that this often isn’t the case. In many IT and product software organizations, testing plays a thermometer role. We ‘take the temperature’ of a piece of software, and let management decide when to ship. In fact I have been told by management, when I’ve tried to be a thermostat, to back off and let them make the decisions. In one case, I have even refused to sign off on a risky release (security-related issue on a piece of beta software), telling the release organization they were welcome to release but they’d have to convince my manager to sign off because I wasn’t going to.

The Testing Thermostat

Thermostats regulate the temperature. In my “Quality 2.0” vision, testing organizations participate in regulating quality. The test organization moves from just taking the project’s temperature, to a role where they are helping to set quality. To do this, testing teams need to:

• Include experienced, respected engineers. At one time in my career, I worked with Christian Hargraves, probably one of the most experienced engineers I’ve ever met. Christian led and maintains the Jameleon project. He’s a sought after expert in the Salt Lake IT market, and has worked for the LDS Church as what I would call a Test Architect. Every developer in the organization knew Christian knew his stuff, and if Christian recommended something, that had weight. Qualified, respected engineers who are part of a test organization can help contribute to quality process and practices across the organization. They can influence core development teams to push quality into their process, from code reviews to adopting best practices.
• Advocate actively about projects. Testers need to be active participants in release decisions, influencing senior management. In many instances, testers are shareholders. In every instance, testers share the benefits of a good release by enjoying continued employment, and in ‘worst case’ scenarios, testers go down in flames along with the rest of the company. In other words, our employment is just as dependent on a good release as any CEO’s. If all we do is report quality, without influencing decision, we are placing our career in the hands of other people. I do not accept this point of view!
• Drive quality upstream: we all know it’s significantly cheaper to fix a bug in the planning phase, early implementation, or even release validation phases as opposed to post-release. Testers need to work to push themselves and their influence upstream—the earlier we can look at code, the better.
• Be involved in planning activities. Take an active role in scheduling. Drive feedback about previous releases (especially lessons learned) into current project planning. If the company has a history of underestimating test duration, speak up about it. Track and present data which shows a more realistic approach. Work patiently but consistently to drive reality into schedules.
• Hold dependencies accountable. If the development team isn’t dropping code early enough for full testing, don’t stand for that! Your reputation as a tester, not to mention your continued employment, depends on a solid release. If the company runs with firm release dates, but development is dumping code in your lap a week before release, you’re going to end up working 90+ hours that final week, and still releasing crappy code! Negotiate and set milestones, and hold development accountable for reaching those milestones. Two things will result: first, your test organization will start to work a consistent, sane amount of hours. Secondly, quality WILL go up because you’ll be looking at code earlier.
• Be involved in establishing release criteria. This is critical—across your business, you need to have consensus on what constitutes release-readiness. If you leave this up to marketing, they’ll say a set of customer stories need to be developed. Leave it to development and they’ll say that, plus possibly that all unit tests need to pass. As QA engineers, we have a unique perspective in that we understand the value of positive AND negative testing. We can help influence the establishing of release criteria which truly reflect product quality. Setting this criteria early, well before a project is nearing a release deadline, removes the emotion and sets a rational basis for declaring ‘Ship Ready’.

These are just a few things a test organization can do to transition from a thermometer to a thermostat. What’s missing? Does your test team take a more proactive role somewhere else? Let’s hear it!

Technorati Tags: quality assurance,qa,software testing,quality

Found a Security Flaw (The Need for Software Testing Everywhere)

Today I was paying a bill online, and I crashed the company’s IIS server (I promise, I did nothing wrong—intentionally). The good news: they write their code rather securely, so they are using parameterized queries rather than embedded SQL statements. The bad news (besides the crash itself) is that they were running with tracing enabled, so I saw the entire stack trace.

I called the company and got in touch with their tech guy. He was really polite on the phone and very open to feedback. I shared a bunch of info with him and thought I ought to document it. So here’s an Open Letter to All Admins Running IIS:

· You need to check your machine.config (and possibly web.config) files and make sure <trace enabled=”false” localOnly=”true” pageOutput=”false” requestLimit=”10” traceMode=”SortByTime”/> See http://msdn.microsoft.com/en-us/library/aa302351.aspx for more info.

· Either spin up Microsoft Baseline Security Analyzer (MBSA) or use IISLockDown and URLScan to scan your server(s) security, especially in configuration. See http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=f32921af-9dbe-4dce-889e-ecf997eb18e9

· You might also benefit from a security review/audit from an experienced, independent consultant.

Security Focus has a decent article on the issue: http://www.securityfocus.com/infocus/1755 (it’s old but still accurate). NOTE there is a chance that running IISlockdown may break something. If your developers built the site with some dependency on what’s actually a security hole, locking down your server could cause issues. I highly recommend that you run this against a test deployment and then run all of your automated tests against that deployment, and that you do some manual testing.

You might also consider picking up copies of

• “Improving Web Application Security” (MS Press),
• “Building Secure Microsoft ASP.NET Applications (MS Press),
• “Writing Secure Code” (I *highly* recommend this – it’s the bible on secure coding, especially in MS technologies), and
• “Secure Development Lifecycle” (MS Press).

From more to less specific, these books are pointers on secure IIS configuration, ASP.NET coding, coding in general, and strategies for maintaining application security throughout lifecyles. The latter book is a great read if you’re on a larger team and/or need to influence management to give you the time for security-related work.

Technorati Tags: software testing,testing,software security