Hi Jim,

I just saw this. Allow me to comment.

>Your last sentence is a true non-sequi
>If the testing is 'difficult' then steps should be taken to reduce the difficulty AND/OR more test time should be allowed to accommodate the difficulties AND/OR the calibre of the testers ought to be confirmed AND/OR more people should be seconded to the testing effort.

Agreed and we do this.

>Everything has a much higher chance of being caught if (non-comprehensive list):
>1) Setting an objective to test 100% of the code and earnestly working toward it even in the knowledge that 'it is impossible' (because it surely WILL be impossible if one starts out with that premise!).

We have a huge (4000+) number of legacy tests that pound on every aspect of the product. Each new or enhanced feature gets a very carefully written and reviewed test plan, specification, and then a battery of tests.

>2) Taking product-knowledgeable people and tasking them with identifying all of the permutations and combinations of conditions that can arise in the product.

That's exactly what we do. As to product knowledge, roughly 80% of the permanent or contract testers have 10+ years with Fox. I think I'm probably the most experienced with 17 years of VFP, FoxPro, and FoxBase but the others aren't far behind.

>3) Taking those same people, and possibly other of less product knowledge, to devise a thorough and comprehensive test plan for each condition identified.

Yep. And each plan undergoes a thorough peer review by someone who is knowledgeable.

>4) Tasking people, under detailed control, to execute each and every test plan and to report all outcomes.

Yep. Frequently.

>5) Addressing each and every test outcome that does not conform to spec.

Absolutely. And we are brutal on those functional specifications to ensure that all possible property values or behaviors are known.

>6) Evaluating the proposed "fix" for poor outcome(s) and ensuring that ANY of the prior test plans that are implicated are tasked to be run again regardless of their previous outcome(s).

I think you're referring to regression testing.

>7) Repeating steps 4 - 6 until the list is exhausted or management decrees termination of efforts.

The product doesn't ship until the Test Team signs off on it.

>You can have thousands of people working on testing a product, but if their efforts are uncoordinated/unplanned and if positive outcomes are not tracked along with negative outcomes then all you have achieved is to have occupied lots of people's time. In terms of quality of the test effort you have NO IDEA if testing was done to 7% or 37% or 87% and that serves no one! Yes, it can be stated that product xxx was the most severely tested product ever, but the truth will be that "severely" is defined as the most manpower expended and nothing else. It certainly saya ZERO about the QUALITY of the testing.

Our tests are extremely well planned and executed. VFP is millions of lines of source code and hundreds of commands, functions, objects, and PEMs. We couldn't ship product if we were uncoordinated.

Talk to Cathy Pountney or Garrett Fitzgerald about our test process - they've been involved with it.