Requirements and the Beast of Complexity

 
Part 1 – The Problem

Roger Sessions, in his thought provoking paper “The IT Complexity Crisis: Danger and Opportunity”[1] , describes an IT industry in deep distress. Roger lays the blame for poor, even catastrophic, IT productivity at the door of complexity. He also acknowledges that ‘some have suggested the culprit is poor communications between business and IT’.

This article endorses both of these viewpoints and promotes a new approach to requirements management that reduces project complexity and improves communication between business and IT. This new approach can be used on its own, or as a supplement or precursor to existing approaches. Critical features of the approach are: detachment of business requirements from individual projects; and the production of testable requirements that can be shown to be complete, consistent, and correct prior to use within the SDLC.

What is a requirement?

BABOK® Guide [2] , Version 2.0, states:

“A requirement is:

1.A condition or capability needed by a stakeholder to solve a problem or achieve an objective.

2. A condition or capability that must be met or possessed by a solution or solution component to satisfy a contract, standard, specification, or other formally imposed documents.

3. A documented representation of a condition or capability as in (1) or (2).”

From Wikipedia: [3]

"In engineering, a requirement is a singular documented need of what a particular product or service should be or do. It is most commonly used in a formal sense in systems engineering or software engineering. It is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user."

These definitions are representative of many attempts to define the term ‘requirement’. We can make some initial observations about a ‘requirement’ as defined by these definitions:

• A requirement is generally assumed to be a document, with use of a natural language narrative implied;

• The requirement narrative could describe any one of a wide variety of loosely defined concepts including a condition, capability, product, service, attribute, characteristic or quality.

• Some concepts in the above, if used on their own, risk being meaningless (a quality? a characteristic?), which implies that there is a larger collective unit, but this assumed unit is not directly addressed by the definitions.

We can assert that there is an implied reliance on natural language rules to aggregate the low level requirements concepts listed above into bigger linguistically valid units, also called requirements regardless of their size and scope. How big can a ‘requirement’ be? A requirement as described above could refer to anything from a clause[4] to a standard the size of War and Peace (e.g. ‘the system must comply with the GAAP’). The use of the word ‘singular’ in the Wikipedia definition (and implied in BABOK) is meaningless when the unit itself is a collective term of unbounded size and scope.

The problem with narrative

To demonstrate how nebulous these definitions are, we suggest that you try taking a requirements document and actually listing each individual ‘requirement’ as per the definitions above (i.e. each quality, each characteristic, etc). This is a surprisingly difficult task. Identifying each individual requirement and ensuring that it is unique within a narrative is a challenge made even more difficult by the ‘idiom effect’ – the idiom effect is the localization and personalization of terms used in the narrative by the author. You are also likely to find that in order to classify individual requirements as proposed, you will end up creating your own idioms – idioms are phrases that have only local, even personal, meaning.

A dearth of standards for writing requirements in narrative form means that requirements specifications (especially ‘business requirements’) are frequently reduced to a series of personal idioms. Most practitioners develop their own requirements writing idioms, which usually change over time, sometimes within a single document. Even when carefully written, English is highly nuanced; ambiguity and imprecision are typical and endemic. When compounded with a variety of personal meanings and interpretations, the majority of requirements specifications are imprecise, no matter how careful the author. Therefore a consequence of capturing requirements in narrative is to create a new class of analysis problem for the downstream developer – analyzing the requirements analysis. And because the requirements document involves some degree of personal abstraction, there will be misinterpretations (the chance of another brain exactly interpreting a narrative requirements document as intended is implausible). This also often gives rise to tensions between the analysis and downstream disciplines, further aggravating the situation.

The network effect

A further cause of requirements induced complexity, and the reason that the larger projects suffer more, is the network effect.[5] The network effect is an exponential growth in the number of connections in a network as the number of nodes increases. For requirements, this means that as the number of individual requirements increases the overall complexity increases at a compounded rate because of a dependency network between the requirements. The term requirement is usually used in its plural form – value can only be realized when each and every requirement in the solution set is eventually built exactly as it needs to be, and that it interacts with or supports its peer, ancestor, descendent and derivative requirements exactly as required. This network effect between individual requirements significantly compounds the requirements communication and complexity overhead.

T Capers-Jones[6] has documented this effect over 10’s of thousands of projects, and provides overwhelming evidence that size is the basis for an exponential growth in cost, time, and risk.

Compounding the complexity through the SDLC

Requirements as described above do not represent a useful endpoint in their own right – the requirements must be transformed into a solution if their value is to be harvested. So the somewhat ambiguous requirements starting point described above is transformed through the series of complex transformations that is today’s Software Development Life Cycle (SDLC). In the typical SDLC there are three major manual transformations required to turn a raw or ‘stated’ requirement into a solution – and as noted, a requirement cannot contribute to utility or value until it is delivered within a solution. The three transformations are:

• The domain ‘need’ or want is translated into a documented ‘requirements’ specification;

• Documented requirements are translated into technical specifications;

• Technical specifications are translated into computer languages.

A fourth transformation, from computer language into executable, is usually fully automated and is in a different class to the above three. The complexity of the traditional SDLC is shown in Figure 1.

Figure 1

These manual transformations share some characteristics:

• Each source specification may be transformed into multiple target specifications and vice versa (i.e. the transformation is often many-to-many);

• No transformation retains reliable traceability between source and target specifications;

• The target specification is typically not able to be fully interpreted by the source specification author – direct manual reconciliation is not normally viable;

• An atomic requirement or specification at any of these levels is meaningless - a requirement or technical specification at any level must be part of a community of related specifications, so that the entire set of requirements must be accurately transformed together as a single unit.

• None of the sets of specifications at any of the stages can be empirically tested and proven to be consistent, correct, and complete (the 3 C’s).

If requirements are the answer, what is the question?

Given that the requirements specification is an essential link between business value and the eventual solution as represented by an operational system, it should not be surprising that the loose original definition of a requirement specification compounded by multiple un-reconcilable many-to-many transformations gives rise to cost, risk, and delay.

This article Part 1 has argued that ‘requirements’ are the Achilles heel of the SDLC, a weak link that in its current form provides a poor foundation for the more mechanical but still expensive downstream development phases. This weakness stems from:

• An imprecise definition of what constitutes a ‘requirement’, and one that is open to a variety of personal interpretations;

• Too much reliance on narrative documentation;

• Assuming business requirements specification to be an SDLC project task – in fact, business requirements are entirely independent of any given project.

• Separation of requirements specification from requirement execution – creating a need for complex handovers that include transformations/translations, and corresponding opportunities for obfuscation and introduced errors.

Returning to our original BABOK and Wikipedia definitions, we can also assert that a well defined and valid requirement should:

• be anchored in a business purpose; and

• contribute to utility or value.

In Part 2 we will discuss how these two requirements characteristics provide a basis for resolving the requirements dilemma – provided that we can address the weaknesses above.