Data Analysis for Business Analysts: Data Modeling in an Agile Team

When ModernAnalyst asked me to write an article on Agile Data Modeling, I suggested that Scott Ambler would be the best person to write an article on Agile Data, but he was already scheduled to write for the issue! So, I spent some time at Enterprise Data World, a conference held in Tampa, Florida, informally polling the data management professionals there about their experiences with Agile.  There were a wide variety of opinions, both positive and negative; everyone was happy to share opinions. Some basic themes emerged regarding the areas where data modeling provides a valuable tool to the Agile team: extensibility, flexibility, and stability. I looked back at the Agile Manifesto[1], it seemed to me that these themes speak directly to three of the twelve principals:

One of the most significant characteristics of an Agile engagement is that technical and business professionals work collaboratively to grow the system. The team agrees upon the goals for the project, as well as the order in which the requirements will be addressed on each of the sprints. They meet daily to discuss next steps, and balance the value provided in each sprint against the development cost and the business needs. At least one team member should have the role of “data advocate”; a person who wears the data hat  and focuses on articulating requirements that need to be assessed using data management best practices in a way that addresses both the business and technical audience.

This article assumes a basic understanding of the agile method [2] and data management terminology [3].

Extensibility - Changing Requirements
As the data advocate, explain the risk or value associated with each data requirement-related decision.   Share these tips that I gleaned from my conversations with the team member wearing the data hat:

• On projects that improve an existing system, the agile approach has the benefit of leveraging existing data architecture. One of the first tasks on an enhancement burn down list [4] should be to identify existing data design and architecture documentation!

• Define terms that will eventually become part of the system taxonomy using a data model. This picture is a very concise way to document a large number of requirements and significantly reduce written text.

• As the data requirements are discovered, capture the information directly in the data model. Use color or font in the model to designate the sprint in which an attribute, entity, or relationship will be implemented.

• Include valid values, data sourcing characteristics in attribute notes and leverage the data model for mapping documentation.

• Annotate the diagrams to highlight existing system limitations and inflexibility.  

• Watch for primary key or relationship changes to existing structures. Tthey cause a ripple effect that needs to be reflected in the burn down list with additional impact analysis and test cases for each piece of code that uses them.

• Explain the technical implications of returning multiple rows instead of a single row through screen and report design.

• Team composition for an enhancement project can be more flexible, because much of the design decisions are a fait accompli; in some cases the data modeler can be an extended team member, called on an as-needed basis.  

Flexibility - Sustainable Development
Maintaining a constant pace is a goal; traditional system development life-cycle approaches typically have peaks and valleys as phase deliverables become due. One data modeling technique enables the slow build-out of detail: the conceptual model. The system logical data model can grow with each sprint; the conceptual model reinforces healthy organized growth. Without a conceptual model, agile projects are inhibited from graceful transitions between sprints by continual rework.

• Include all potential requirements in the conceptual model in the very first sprint, but only build out the logical model detail by sprint as necessary.

• Incorporate history requirements in the concept model to minimize impact to key structures.

• Include internationalization and localization [5] requirements in the concept model for the same reason.

• Remember to define a primary key and relationships for each major concept in order to provide the framework for the data design.

• Use meta data to document policy decisions on data sourcing, appropriate usage, and security classification.

• Recognize in the design, that data structures have a long life, and that compromises made in structural design will impact any future consumers of the data. DOCUMENT COMPROMISES in the data model.

Stability - Technical Excellence & Good Design
For a project team to succeed using the Agile method, each team member must share their practice knowledge, and be honest about their skills. Every development project has a data component, even if only reading and displaying information. The danger in an Agile project is to focus on development speed and budget constraints, at the expense of maintainability.

• Identify data governance as new structures are created to establish clear responsibility for future change decisions.

• Implement the maintenance of reference data before the master data to ensure business ownership of code sets.

• Elevate requirements that establish or change architectural direction to management for review.

• Follow all documented standards for coding, naming, change control and security. In most shops, they were developed to reduce business and technology risks.

Conclusion:
These lists are only a small portion of the data management best practices that can be applied. Search out the data management professionals on your staff and see if you can identify even more.

If you are on an agile project,
what hat do you wear?

Related Data Webinar: Webinar: Got Big Data? How can you combine the Big Data with traditional enterprise data?

Resources:
Scott Ambler - http://www.agiledata.org/essays/agileDataModeling.html
Bill Lewis - http://www.tdan.com/view-articles/5000

Author: "Loretta Mahon Smith, CCP, CBIP, CDMP, is President for the Data Management Association - National Capital Region, a member of the Board of Directors for DAMA International, and an ICCP Certification Counsel Member. She is a 25 year career Data Architect in the financial services industry."

[1] http://agilemanifesto.org/principles.html

[2] http://www.agilealliance.org

[3] Mosley, Mark. The Dama Dictionary of Data Management. City: Technics Publications, LLC, 2008. ISBN: 0977140040

[4] http://www.controlchaos.com/about/burndown.php

[5] http://en.wikipedia.org/wiki/Internationalization_and_localization