Must software requirements precede testing?

In a recent online IBM jam,  I read a statement in one post that struck a nerve in my cynical mind. This post stated that: “You cannot (or should not) write a test script before you have your requirements.”

Perhaps this is sometimes true but it is frequently not true. Let me provide a counter-example from a Mil-Std development where the default perception is that requirements must come first. When I was the chief-software architect on the CCPDS-R project (a $300M Command Center for Missile Warning), we had a very explicit set of “performance requirements” to be met. One of these requirements was that all of our missile warning displays would be generated with a 1 second response time. That sounds clear, crisp and measurable.

We employed a very iterative, measured, demonstration-based approach to this project and by the time we got to the CDR we could demonstrate most of the 50+ displays. [CDR in the Mil-Std world was roughly the project midpoint representing the transition from the design phase to the production and test phase.] The users (Cheyenne Mountain Operations officers) were ecstatic with the early capability that we demonstrated and they had collaborated closely with our team to get to the display formats they preferred. About 10 of these displays, with complicated graphics and some complex data synchronization challenges, required about 1.4 seconds to generate. The MITRE technical monitors for our USAF customer demanded a plan for how we would get these 10 displays to meet the 1 second response time. At CDR we told them that we were confident that we could meet the requirement, but to get there, we needed to freeze the display formats and remove some of the “flexibility” we had built in so that we could optimize performance, a very straightforward tradeoff.

It was at that point that our user stood up and said “wait a minute, we like the flexibility, and we like the responsiveness, we would prefer to interpret the requirement as 1 +/-.5 seconds because the specification requires 1 second response, not 1.0 second response.” Wow. As late as CDR, we had our user arguing for us to ease off the specification. Why? Because that requirement was a rough guess at quantifying the word “responsive”. They had not questioned that quantified guess (now laid in stone in a contractual requirements specification) until the test case had been written and executed so that there was an objective basis for judgment rather than the subjective “1 sec feels about right” speculative basis.

On this same project, we were also required to run all our software under a “double JCS scenario” without consuming more than 50% of the computing resources to ensure there was platform capacity for future growth. This scenario was roughly double the worst case expected in live usage. Until our test team codified the message sequences that would be produced under such a scenario from all the external sensor systems, the user and the customer really had only a surface understanding of the ramifications of this scenario. The act of codifying this scenario in a test case resulted in an enlightening, and mutual understanding of the practical meaning of “what” they wanted to require and how to quantify the specification.

Lessons learned:

1)  Designers and analysts and modelers and requirements engineers have entirely too much freedom to create and generate “speculative” documents and models that are still loaded with uncertainty. Testers come at things from a much more realistic and constrained world. The combination of these two perspectives resolves uncertainty faster and earlier.

2) Requirements is one of the most misused  words in our industry. Very few things are truly required. Most things are negotiable. Our understanding of the problem space evolves and changes as our understanding of the solution space and the constraint space (plans, economics, realities) evolves.



Be Sociable, Share!

Tags: , ,

3 Responses to “Must software requirements precede testing?”

  1. Jim says:

    Corollary: you’ll do better if lots of room (time) is left to respond to any negotiation once your client realizes that he wants (needs) to negotiate (change!). Things I’ve been heard to say (certainly not original): (1) “Embrace change” (2) “Requirements aren’t”.

  2. Dave says:

    Stakeholders may not be able to provide effective requirements for a novel system until they engage with an approximation of that system, ideally a malleable approximation that can iteratively evolve to a production-quality deliverable.

  3. Tim says:

    I agree with all the points and comments. But you did have “requirements” (I use that loosely here) before you started testing. You had something to shoot for. Everything else said is completely true, especially projects that create new technology and push beyond anything done so far. Military and Aerospace projects tend to do that.

Leave a Reply

CommentLuv badge