I wrote an algorithm in Real Basic a while back and now under Xojo it doesnt work. Not because Xojo broke it, but my algorithm is broken. And I didnt notice it way back when. When I pull out of the archives an old copy of Real Studio and compile it, it breaks just like Xojo does. My regression testing (Unit Testing) must have failed me (I must have not written all the correct tests).
Now I get to scramble on how to fix the algorithm and retro-fit every app that uses it. YIKES! sounds like I will be having fun for a while.
Now I am off to the ATL Xojo meeting before it dumps slush (rainy ice) on us tonight. I am going to use this to clear my mind before then design binge.
This is one of the things that keep QA engineers up at night. Does the test suite test the code as written or does it test the code as it is supposed to function? All too often, especially in unit tests, the test is written to see if the code works. That approach leaves three questions unanswered, 1) does the code do what it is supposed to do, 2) is the test testing the right thing, and 3) is the test itself functioning properly.
Rick’s approach is actually the correct one. Tests, especially after the initial unit tests, should not be testing the code to see if it works, but, rather, to see where it breaks. We assume, sometimes wrongly, that the coders wrote the code to do what the specification or the customer says it should. Tests should be about 10-15% testing that and 85-90% testing for where the code breaks. Customers generally don’t care how the code works. They care when it doesn’t.
I have a cousin who was a lineman for the major electric utility in southeast Michigan. Any time that he had to attend a meeting where they were going to teach the linemen how to use some new computer system the first thing that he would do once he saw the system was running on the screen in front of him was to depress as many keys on the keyboard as he possibly could. He would do the same for any input field that he would encounter on the various screens. His first attempts often were successful in bringing down the application. However, it was also a learning tool for the programmers. Test for idiotic input from the users. Usually bad characters being entered are an accident. In his case it was a challenge.
Reminds me of my old boss. He had an uncanny ability to find bugs and edge case scenarios. Once, after a couple of months and several tollgates we presented to him an application aimed at tracking projects and resources for customers and the first thing he does after logging in is click on “create new project” and proceed to keep the “ñ” key pressed for half a minute in the “title” field. He then goes and tries to create the project like that.
The program crashed so hard we didn’t even have time to gasp. He didn’t say a word, we just took our stuff and went back, to tighten all the validation code all over.