The damaging consequences of organizational amnesia, and how BAs can help prevent and remediate them

After mapping out the as-is process at a high-level, the BA’s stress level starts to go up. “There are three complex modules in this system, and so many details about the as-is state that I still don’t understand! The legacy system barely has the original requirements documented, with plenty of change requests implemented later without proper documentation. How am I supposed to finish my deliverables on schedule and without mistakes given my limited knowledge of how the system being replaced works?”

Soon the BA starts to have nightmares in which the new system goes to production and the next day an entire group of users complain about the loss of a critical feature required for doing their job. An agile process is established for the incremental delivery of the solution, and three months after the project is completed, disaster strikes: the company has to pay a hefty fine because of a violation of state and federal rules that prohibit certain kinds of products being sold to certain kinds of customers.

Because the manager who understood the applicable regulation had already left the company, the circumstances leading to this type of non-compliance are rare, and the logic to prevent the issue was buried in the code of the legacy system, the business was exposed to avoidable fines and legal costs. And, after all his hard work, the business analyst was blamed for the requirement omission.

How to avoid this no-win situation?

The best way to avert his type of crisis is to prevent "organizational amnesia" in the first place. In other words, make sure an organizational knowledge base is created for systems supporting any mission-critical process, with updates applied as changes happen until the solution is finally retired.

You may be thinking, “While this is the obvious answer, it’s easier said than done. Everybody is busy, documenting process and system behavior is far from the first priority in our team, and too much documentation only means there’s more content for people to ignore because there is never enough time to read everything.” But while these objections have merit, they are not insurmountable. Even if you don’t think prevention is feasible in your organization, many of the preventive tips included in the section 1 below can also be useful when you’re in a situation in need of corrective action, addressed in section 2.

1) Preventing organizational amnesia through adequate system and process documentation

Objection #1: Everybody is busy, and documenting system behavior is far from the first priority in our team.

Solution: Adjust your reward system, and establish a process to support the goal of creating and maintaining an organizational knowledge base for each system supporting a mission-critical process.

The first step in the fight against organizational amnesia is to change your reward system. Make part of the criteria for the performance review of your business analysts, project managers, system analysts, and software engineers, their contributions to the organizational knowledge base. Managers should encourage their directs to dedicate part of their time to creating relevant documentation for mission-critical systems, and make it clear that this contribution will be taken into account during performance evaluation.

Along with appropriate incentives and rewards for the desired behaviors, it’s important to have processes in place to ensure that the right documentation is being created and updated on a regular basis. Examples:

• An agile team may reserve a portion of their retrospective meetings to go through a checklist to see if there’s anything important to add to or replace in the organizational knowledge base as a result of the work just completed.
• As part of a prototyping process, the project manager may be asked to document the answer to the question, how well did the proposed solution work, what was missing, and what was done to fix it?

Objection #2: Too much documentation only means there’s more content for people to ignore because there is never enough time to read everything.

Solution: Use relevant criteria to define what “quality documentation” means, specifying what kinds of information need to be captured and categorized, and in what level of detail.

To be usable, knowledge needs to be documented at the right level of detail and organized in a way that makes it easy to find and consume. Ideally, the organizational knowledge base will be kept in an efficient collaboration tool such as an effective wiki system. The selected tool should make it easy to organize information using a structured hierarchy, and include a powerful search engine to help the team find information quickly and easily.

In addition to requirements specifications and business process diagrams showing the flows between related actions required to achieve a valued result for the organization, the knowledge base should include the following elements for any mission-critical system:

a) Any deviations from the original user-facing requirements in the final implementation, even if small. For example, “The original specification stated that users should be able to select from: and to: dates for the sales report without establishing any restriction; due to our data retention policy of 12 rolling months, the from: field will only accept past dates within that 12-month interval”).

b) Any decisions that affect the user experience that had a dissenting opinion. For example, “There was a back and forth between the heads of marketing and sales as to which department would have permission to qualify a lead (rank prospects against a scale that represent their perceived value for the organization). In the end, it was decided that only managers in the sales department would have access to the lead scoring functionality for reason X."

c) Any unusual user-facing behavior the system has compared to similar applications. For example, “Traditional CRM systems typically allow a one-to-many relationship between prospects and contacts, as the same prospect may have multiple contacts associated with them. Due to our internal policy of only discussing our offering with a single designated contact from each prospective customer, our CRM enforces a one-to-one relationship between a prospect and a contact.”

d) Any user-facing changes applied to the system after the go-live. For example, “Release 2.0 changes the scale to rank prospects because of Y. Archived leads were not affected, but all active leads were modified to reflect the new scale. Appendix A shows a comparison between the old and new scales used to rank prospects.”

Note that in the examples above the underlined words represent the “why” behind a constraint, decision, change, or anomaly. This is the most neglected piece of information in system documentation, and one of the biggest causes of failure or cost overruns in projects to enhance or replace a legacy system. Without the context and rationale for previous decisions, it’s practically impossible to distinguish between the aspects of the solution that must be preserved in the to-be state for legitimate reasons, and questionable decisions that if maintained will only diminish the value delivered by the new system.

Knowing not just what, but also why the current practices are in place is critical for a business analyst to make the right decisions during requirements discovery during an enhancement or replacement project. Armed with this information, a BA can validate with key stakeholders whether the rationale used for the past decision remains valid, or the organization can benefit from more efficient practices in the new system.

Note also that all the elements described include the qualifier “user-facing” or “user experience”. Being selective about the content of the organizational knowledge base is important to increase its utility for analysts and decision-makers. Details like user interface design standards, architectural conventions, how-to user guides, while important for enhancement, training, support, and maintenance purposes, don’t belong to this type knowledge base. Documentation should be limited to information about the business problem being solved, the business rules involved, the user-facing properties required of a fit-for-purpose solution, and the reasons for those requirements to exist.

For example, “A user must be able to archive leads in order to remove the clutter and ensure that important leads are easy to find and follow up with" is a user-facing property that solves a business problem (increase visibility of important leads), and therefore should be part of the organizational knowledge base. “When a lead is archived, the system updates lead status in the database with value = INACTIVE” is an implementation detail that wouldn’t be included.

The principle behind this approach is called “separation of concerns”. While it’s important for the programmers doing maintenance on the legacy system to know that, behind the scenes, archived leads are identified by an attribute called “status” with value “INACTIVE”, for the goal of organizational knowledge preservation this is not relevant information. If in the new system the user-facing functionality is preserved, while in the back end the archived leads are identified by setting a flag “Active?” to “No”, there would be no difference in the value delivered by the solution.

When talking about user-facing behavior, however, even a piece of information that seems obvious in the moment can become invaluable in the future. Consider the case of the constraint of one-to-one relationship between a prospect and a contact (the example provided for “type c” information --unusual user-facing behavior). Without a documented explanation for the restriction (compliance with a valuable internal policy), stakeholders not involved with the previous decision might assume it was just a poor design choice rather than a capability required to fulfill a legitimate business need. The requirement could be easily overlooked during the specification of the new system, with negative consequences for the business.

Often, “type d” information (user-facing changes applied after go-live) arises as a reaction to some past disaster. The logic that existed in the legacy CRM system to prevent selling certain products to certain customers might have been implemented after the company faced a significant monetary loss for violating federal and state regulations. As you work on a replacement system, without an organizational knowledge base to consult it would be easy to neglect the logic created to deal with that specific, rare situation--until the next catastrophe makes its original reason for existence depressingly apparent.

Since most BAs will find themselves in the position of having to deal with "organizational amnesia” from time to time, it’s important to talk about how to handle this situation when it can’t be avoided. Besides doing problem interviews with stakeholders to understand the business need, and mapping out the high-level “as-is” process, there are three additional steps that can help mitigate the risks of a legacy replacement project implementing a solution that fails to deliver the expected value:

I) Enlist the help of users, software developers, and other people with institutional knowledge to build your own “organizational knowledge base"

Ask people about the elements that belong to the organizational knowledge base (described in items “a” to “d” in section 1):

“Do you remember any contentious decision made when the system was initially specified—anything that generated diverging opinions, such as who should have access to a certain feature, or for how long should records be retained for reporting purposes? Anything that had to be changed from the original requirements due to a technical constraint or the need to comply with external or internal policies? Any unusual behavior we wouldn’t find in similar systems but it made sense to implement for our company for competitive or compliance reasons?”

Using the “separation of concerns” principle, document the relevant information received from this audience.

II) Investigate more deeply the knowledge areas that tend to create problems whenever a legacy system is being replaced

Certain aspects of the solution are more likely to be neglected when a legacy system is being replaced, often creating serious problems that prevent a smooth transition to the new system. In addition to performing the interviews mentioned in step I, look for things like:

• Explicit or implicit business rules that should be enforced by the system. (E.g., "A prospect may only be associated with one sales district.”)
• Permission rules that must be preserved. (E.g., “Sales reps can only view and edit the tasks assigned to them.”)
• Special filtering or sorting features that need to be available for slicing and dicing data. (E.g., “When a manager opens the task list, she should be able to filter tasks by team, region, customer, or stage in the sales process.”)
• Special considerations regarding data retention. (E.g., “Archived leads will remain in the system for one year; customers records will be retained for five years.”)
• Special considerations regarding time-consuming queries. (E.g., “To prevent performance issues, when a sales rep searches for leads, limit the results to active leads unless the rep opts to include archived ones.”)
• Special considerations regarding the cardinality or hierarchy of objects in the system. (E.g., "A prospect may have only one contact assigned to it.” "Client companies may have a parent-child relationship with other companies.”)
• Are there any quality attributes (performance, reliability, scalability, security, interoperability, etc.) that need to be preserved or enhanced? (E.g., “Calculated sales quotes must be displayed within three seconds of completing entry of the service type and count.”)