Here is an idea I've been considering for while: how to get more mileage from your unit tests. Unfortunately, most unit tests rely upon a static set of inputs, in which you test the output. While this provides a good basis for tests, it by no means is an extensive test of the system. People familiar with real-world user interface testing will find that users will misunderstand intents, mistype, and sometimes, maliciously attempt to break the system. How will the system respond during these situations?
The typical testing process involves creating a few boundary condition tests, running those, for each release candidate. If the user acceptance testing happens to find a case where data inputs created an unexpected result, the discovered test conditions are added to the set of unit tests. Although necessary, this is a slow and arduous process. This can be used in conjunction with StochUnit to get to those boundary conditions (hopefully) earlier in the process, which should have lower cost overall.
Each developer on the team and possibly your CruiseControl server run tests many times per day. What if each test could get closer to boundary conditions when tests were performed? Short entries, long entries, non-alphanumeric entries, etc. In any situation your application needs to recover with some degree of fault tolerance. Your software system is a model of the ideal theoretical business application system. If you want to test a model in the world of science, you test it with a variety of inputs. These fall under explicit input definitions: sensitivity analysis, and stochastic or Monte Carlo methods. StochUnit allows access into a larger test domain based on Monte Carlo inputs. From the Wikipedia entry:
There is no single Monte Carlo method; instead, the term describes a large and widely-used class of approaches. However, these approaches tend to follow a particular pattern:
1. Define a domain of possible inputs.
2. Generate inputs randomly from the domain using a certain specified probability distribution.
3. Perform a deterministic computation using the inputs.
4. Aggregate the results of the individual computations into the final result.
StochUnit is a package that will help you get additional results from your unit tests with minimal effort. Existing tests may take some converting over to take advantage of this method, but once that is done, you have more confidence in the resiliency of your application under a range of inputs.
The first step is to convert your existing tests to run with property files instead of hard-coded data. The next step is to have these property files get generated. This is where StochUnit comes in. You provide a basic set of properties (basically to insure that you test your basic condition), and a second set which is determined stochastically based on settings you create. You can still go back to running with the basic properties or the dynamic properties at a later date.
This concept applies to unit testing in different programming languages, but first I will focus on Java and investigate others as time allows.
References
http://www.slideshare.net/timmenzies/on-the-value-of-stochastic-analysis-for-software-engineering-1739199
Followup:
2010/12 added to sourceforge
