JimFosberry

(** Fair Warning – this is a post about Software Development, so it may not interest folks not involved with the industry**)

Some personal thoughts on Software Testing an Quality Management….

I’ve been in the Software industry for nearly 20 years at this point, and have come to appreciate the value of creativity and understanding the user more and more. Personally I have seen many products suffer from a lack of control and balance in development, but even more seen products suffer from stagnation of process and management for control – sometimes stated as predictability of delivery.

There is a need for process to support the development of software within a team or organization. The business side of commercial software needs to know when revenue will be recognizable on the investment made to create the software. Until you have users using the product, you cannot recognize revenue from the sale of the product, so you have to have a target to make the product available to the user community – that’s the hard accounting fact of life.

Process provides the methods when starting a project to determine what the product will be (features, requirements, etc.) – knowing the content provides the ability to set timelines and targets for delivery. This is how and where the business determines if the project is workable in terms of recognizing the revenue and achieving a return on the investment in producing the product.

Once the project is initiated, to ensure the deliverables are on track to meet the estimates, there needs to be some method of assessing where the project is at a given decision point. Were the estimates (guesses) about how long particular tasks take proving to be somewhat accurate? Knowing this is always important – if you don’t know you are behind or ahead of schedule before the end of a project, you cannot adjust the project expectations or locate additional resources to help right a slipping schedule.

These are hard truths in Commercial Software Development. The organization is trying to sell the product, and needs to spend the development cost up front. There are not unlimited resources in terms of hours or effort, so the scope of the project (what’s in it) is likely the only thing that will be negotiable after the project begins.

Thinking about the Quality profession, and the role of a Quality organization in the delivery of products to market, two things are strikingly clear to me as a practitioner. One is that the Quality Assurance and Testing efforts don’t make a product have or lack Quality. The other is that the Quality Assurance practitioner has to have a clear focus on the user and purpose of the project product, and have the ability to creatively execute quality practice to meet goals.

Traditional waterfall development passes completed code to a tester who runs through various scenarios and identifies defects. Unfortunately in this method, finding a significant defect jeopardizes the delivery of the product on schedule. Often the stress of finding a critical defect late in a testing process creates the appearance that the tester did not find the problem soon enough and caused the risk to the schedule.

Its actually a hard argument for the tester in the waterfall method to defend against. Shouldn’t the most critical areas always be tested first?

Looking more closely at the waterfall methodology the limits are clear. With testing at the end of the cycle, if any other component of the project slips – lets say development takes longer after the design ran a little late – when you assume the end date isn’t moving, the last man in line is going to be pinched – so there isn’t as much time to test. During the phases leading to this point, the statement repeatedly is made the ‘we can make up for this delay as we move forward, I am sure.’ – and pretty much everyone else can think that except the last man in line…..

Regardless of the methodology in use, the testing and quality organization cannot create the attribute of Quality in a product. All the Quality organization can do is measure the relative quality of a product based on comparisons of like efforts in the past. The number of defects at a given severity level over time, the mean time to defect in test, the coverage matrix of a given effort and percent completed are all aspects of this determination, but all of this is looking at what has been delivered, not the means of creating better quality software. From a testing perspective (which typically is an aspect of a Quality program) the goal is to find all critical and as many of the non critical defects in a product as possible. Again, this is in a product that has already been coded and delivered. The tester can’t prevent a defect from existing at this point.

The point is that the Quality Assurance discipline and practice is focused and targeted on defect detection and remediation. The creation of a product with the attribute of a certain level of Quality is owned by all disciplines – from conception of the project through the delivery – Coding, Design, Requirements, Delivery, Support, Training etc. all play a part in achieving the accepted definition of Quality within an organization.

Having had the good fortune to work with several development and engineering organizations over time, I have found that the definition of ‘Quality’ is a wide and varied area. Each organization and in many cases, each discipline has its own working idea of what a Quality product is and looks like. The quality organization is always working within this backdrop. If development defines the Quality as being a product with no Sev1 or Sev2 defects outstanding, then that’s the goal of delivering the product. The definition can be broadened to the unattainable level as well - a particular party could feel that a product is not of sufficient Quality until there are no outstanding defects.

Personally I don’t think there is a correct or right answer to what is a Quality Product. In each situation the relative quality of the product is measured externally by the client, not by a pre release product development organization. Internally by focusing on the intended use of the product and the resources in man hours and calendar dates, decisions are made to find the Quality definition that fits for a product at a given point in time.

For younger products, realizing the business need to have a return on investment for the product being introduced is an important driver of all decisions. This includes meeting the agreed upon or set delivery schedule to GA. This may mean that there is an acceptance that the product will have a higher rate of defects known, and after delivery, will have critical defects coming in from the field. For more mature and stable products higher expectations of lower defect counts and fewer or no critical defects coming from the field after deployment will be the expectation.

I think of this as compromise in the positive sense of the word. If you don’t ship new functionality, you are missing key market demands of your product and will lose client share. If you continually ship products with multiple critical failures, you will also lose client share when customers seek more reliable solutions for key processes. Often an approach is to have scheduled ‘Updates’ – i.e. bug fix releases – planned prior to shipping at a near interval (ship date + x weeks) to have in place a target for responding to early adopter issues.

If you never ship the new functionality your organization will continue to accumulate cost with no matching revenue to off-set it, and the organization will fail. A senior industry person once advised me that asking for more time to test is not a good approach to the time crunch at the end of a project. If you are asking for more resources (time and man hours) you are asking for more cost, and the question is what does the organization get for the additional cost? The temptation is to say fewer unknown defects and/or fewer critical defects reported in the field. So how much time do you need to have no unknown defects and no critical defects in the field? If you get one more man week will that eliminate 20% of the critical defects?

It’s an exercise in trying to measure an event that doesn’t happen. We don’t award accident avoidance; we tend to reward accident response much more readily.

As a tester, the focus is on the functionality that is changing or being added to an existing product. There are also regression of areas of the product which are directly impacted by these changes, and some core function testing that has to be done on-going to prevent the undetected negative impacts on critical points in the software (i.e. if you add a feature that makes the user’s life much better, but the log in screen doesn’t work, they won’t realize any of the benefit.)

When testing I try to remove any moral or judgmental elements from the execution of the tests. It isn’t ever possible to achieve 100% coverage of a product. If you look at a simple binary flow of decision making involved with the testing of a product, the exponential growth of possible use cases means you will not have sufficient time to test all possibilities. Even if you had unlimited time, the value of executing all of the cases is not there. Focusing on the cases that users are actually expected to use has a clear value. When looking at what the management of a project sees – when testing and representing a test effort I try to show the decision and focus is driven by the Use Cases that map to intended and expected use of the product. Project Time is not well spent when it is locked into a corner expending resources on unrealistic and not expected cases. Even if a defect is found in this type of testing, the likelihood of it being experienced by an actual user is too small to justify the investment.

When preparing test cases to support a Quality goal for shipping a project, always try to anticipate the question “Why are you running this test?” If you can’t come up with an answer that satisfies you in the hypothetical, the case probably is not as urgent as a case that you can answer that question for.

When looking at the focus on the Process of delivering software solutions my own opinion is that following a rigid and standard process for multiple products is generally not a successful approach. One case in point is the methodical focus and measurement of Test Artifacts. While having standard artifact templates available is useful to the tester, the expectation that all artifacts are applicable to all projects, or that the preparation of the artifact is a sign of a project that is on the path to success has not been proven out in my experience. If a tester spends the more time documenting test steps than executing test steps, the value of the testing effort to the project at hand is being undermined.

There is an art to testing. It is not a strict science. You can’t say I executed 5000 test cases so the test coverage is where it needs to be. If the tester is limited in their testing to the cases that are prepared, major problems tend to develop as the tester feels they are not empowered to test what, during execution, seems to need the most focus.

We map the Test Plan and Test Cases that are prepared formally back to requirements (and some organizations opt to map them to designs). This is like a map for a trip. You start out intending to follow a certain path to arrive at a destination. Perfect world that’s the path we actually take, and on occasion that may actually happen. If en route you get hungry, need to use the rest room or need fuel, you stop worrying about the anticipated itinerary and attend to the immediate need. If the road has a detour, you don’t try to stick to the path through the construction, you follow the detour. If traffic is backed up due to an accident, and you know an alternate route, you may take it and pick up on the planned route later in the trip.

Testing is the same process. When executing a Plan, you may find the product being delivered has changed since the initial design and your plan no longer matches. You may find that while testing a particular area of the product ‘feel’ less stable than others. Even if you planned to spend the same amount of time in resources between the two, you need to shift priorities and work on the less stable point. There is an art to these adjustments and changes of direction. Quality assurance and test organizations that are successful should always be looking to reward the ability to take initiative and adjust in these situations. Don’t worry about changing the plan documents; just make sure the testing is being done on the areas that need the testing.

Testing is a collaborative process. No discipline has all the knowledge or pieces of the puzzle for delivering software. Planning what areas to focus on at the start of the effort or test phase with the development and product managers, designers and engineering functions shows a willingness to accept input and builds on team strengths. Having the support of the full team, and providing rationale for proposals helps to ensure all parties are focused on the same use cases and impacts. If the Developer knows what you are testing for, they are more likely to deliver functionality that meets the expectation of the test. Similarly, if a case identified by the tester is superfluous, the team can identify it and convince the tester before time is wasted on cases that are not going to provide payback. Open discussions give all areas an ownership of the testing effort.

When looking at project metrics, any program that focuses on the percent of planned activity completed is missing the point. If there are unplanned activities that are being completed that are found to have a higher priority than some activities that are planned, the accomplishment of the higher priority tasks is a good thing, not a ‘risk to the product delivery.’ Generally the tasks that are introduced outside the plan replace tasks initially planned. Even if they don’t, the product is better served delaying the completion of lower priority items after the product is shipped prior to the first service pack than to not accomplish the undocumented higher priority tasks prior to ship.

Looking beyond waterfall, Agile methodologies function on this basis as a primary idea. Constantly adapt to the build being tested and the changes being delivered in the time-boxed drop. You aren’t receiving a final complete build, so regression is limited as well, and the test is time-boxed so focusing on the important cases is vital. Documenting a case that necessarily will only be run once in an iterative build is not a good use of resources, even if the case is a priority for that iterative build. The speed of change and open collaboration require the tester to be able to shift gears and meet the spirit of the goal of the testing much more than executing a rote set of steps.

Looking ahead all indications are that Agile Methodologies are becoming prevalent and replacing waterfall approaches rapidly. Agile itself is a broad set of methodologies and approaches, but the tester and Quality practitioner of the future will need to be able to function and adapt to the approach being used on a given project to remain successful and relevant. “Why am I running this test?” is a question to ask repeatedly and continually to support confidence in the engineering collaboration that is key to Agile methods.

From my current perspective, the flexibility to understand the User goal in the features being delivered, and the creativity to adapt to changing situations in the development cycle are the keys to supporting higher quality software. Management by controlling units of effort is not effective, in my opinion, though understanding defect trending does show more insight into coverage and readiness for market, both leading to and after market delivery.