Understanding and Managing Agile Transitions

Software development process is undergoing seismic shift from traditional waterfall software methodologies towards agile methodologies. Agile software development methodologies deliver high quality software products in rapid iterations with high flexibility and adaptability to changing conditions. This article discusses the dynamics of agile projects by comparing it with the SDLC project framework to help the IT leaders and organizations plan effectively for transitioning to Agile software development methodologies.

I. INTRODUCTION

Agile project methodologies have proven to deliver predictable results in changing business conditions where the IT managers are presented with limited budgets and stringent timelines. For projects that have a lot of moving parts and technical complexities, it becomes difficult to plan the project from start to end using the traditional plan driven approach. With changing business priorities and market conditions, the change driven approaches have proven to offer flexibility and adaptability to changes and deliver predictable, high quality working software products in rapid iterations and of high business value.

IT leaders and organizations need to choose their development methodology with a long term vision in mind and therefore need to consider how the long term vision will help to remain competitive in the market and capture a broader customer base with high quality products that have high value adds to the customers. Organizations also need to establish a middle ground to find a perfect balance to manage the project management and software development side of these agile projects.

Agile methodologies focus on informal methods of project management and emphasize on flexibility, increased communication and transparency. The methodology also promotes less control as exercised by project managers in the traditional approach. As much as the change driven approach promises high quality output, it has its own share of adoption hurdles. Often the Agile methodologies are viewed as being chaotic, with lack of project control and might appear to be developer centric so the methodology does not get the management buy- in as the role of project managers appears to be reduced.

The adoption of new software methodology should be relatively easy to implement as organizations cannot usually halt their current projects or slow down new development projects to learn a new methodology. (Krueger 2002) The transition usually happens when the driver for the change is a turbulent business environment with constant changes that recognizes the need for agility in their projects. (Nandhakumar and Avison (1999)).

II. BEGINNING THE TRANSITION TO AGILE

Traditional plan driven approach emphasizes on having all the project requirements defined early in the project and locking down any new requirements or scope creep by a sign- off process. The involvement of business owners is seen predominantly in the early stages of the project and then during the user acceptance testing phase towards the end of the project. Lots of ceremonies and documentation are part of the traditional model, limiting the feedback loop for identifying scope of improving the existing process. What prevents many organizations in adopting agile methodologies lays in their lack of understanding in the alignment of the processes in their traditional software development methods with the agile methods. The sections below explain the various phases of the traditional methods as aligned with agile methods.

A. Project Initiation Stage

The Project initiation phase defines and refines the vision of the project. The vision basically sets a desired end state and the software development process will be defined around it to achieve the end goal. The project schedule will result as an output of the initiation phase. Other artifacts referenced in the adaptive development cycle, the vision charter, project outline and product specification outline may also be produced as output of the initiation phase. (High smith 2000)

In this phase, after the PMO has approved a project and identified the project manager, it becomes the responsibility of the project manager to move the project ahead. With the initial list of stakeholders identified, the project manager will be involved in setting up the kick- off meeting for the project.

The goal of the kick-off meeting is to discuss the scope of project work and identify the “out of scope” items, project trade-offs (Scope vs. time), etc. The overall project risks are identified and an initial risk register is created. The project manager is also tasked for performing an in depth stakeholder analysis and to establish a RACI matrix. The business stakeholders are introduced to the concept of a “product owner” and a product owner is selected by the business owners from amongst them.

The output from the kick off meeting can also include

• Scope Statement

• Understanding of business case & value

• Identifying any dates to meet for regulatory or compliance requirements

The project manager and business analyst would meet with the business stakeholders to derive the list of high level features and possibly user stories for the release planning session. Schwaber and Beedle (2002) recommend that this collaborative initiative with the customer for creating the initial product backlog can result in several days for a project adopting Scrum methodology for the first time so time needs to be built in for this requirement gathering effort.

This is similar to the requirements gathering session of the standard SDLC process but without the same level or formality or details into the features. The Business Analyst and the technical team members meet to discuss the high level project requirements. This is necessary for the technical team to start think on architecture, design, and perform feasibility checks, etc.

B. Release Planning Meeting

In the release planning meeting, the project team (application development team, DBA’s, technical architects) are invited and introduced to the business team to discuss the details of the project from the implementation standpoint. This can be related to the JAD sessions or focus groups where the desired product features are discussed in more detail.

During the release planning meeting, the team members (business and Scrum team) are introduced to each other either via an ice breaker exercise or through formal introductions as applicable to individual organizations. Meeting agenda is handled via the Scrum board to introduce and expose the business team to the Scrum board concept and how the tasks move across the board. The details of the in- scope items, out- of- scope items, stakeholder list, assumptions & constraints are displayed to be viewed by all the attendees. The business analyst can explain the details of the project and all the features for the benefit of the team to establish a set level of understanding. The technical team can present the proposed architecture or design of the project if available.

The business analyst, along with the Scrum master, organizes and visibly displays all the features and user stories related to the features on the table or visual board. The business stakeholders are tasked with prioritizing all the features in the increasing order of priority from left to right. When the features are prioritized, the user stories below them also move with the feature. The product owner, with the help of the business stakeholder, can then identify, isolate prioritize the user stories from within the prioritized feature list which has the highest business value and needs to get done as part of this project. This helps to identify the stories or features that have the highest business value and without delivering them, the project would not be deemed successful or complete.

In this planning session, any gaps in requirements and dependencies are identified and new user stories can be created to cover the gaps. The technical architects along with the rest of the technical team identify all the features and stories that have high risk for implementation or may pose technical complexity. These stories are prioritized to be handled earlier in the project. High risk items are handled earlier in the project to mitigate the cost of project failure. Any technical code analysis (spikes) or pre-work (technical debt) are identified and discussed. They might also potentially end up as user stories plotted on a roadmap.

All the prioritized user stories are then moved separately from the rest of the group and are identified to be delivered first as a part of the project. This group of user stories is called as the front burner group. The rest of the user stories that are not part of the front burner are also prioritized and become the back burner group. These are the stories or features that have business value but not a high business priority to get implemented earlier in this project. The stories prioritized in the front and back burner make up the product backlog. At the end of the release planning session, a list of prioritized user stories or features are generated.

C. Estimating User Stories

The business analyst, technical team, Scrum master and the project manager participate in the estimation session. The product owner is optional for this meeting. The team agrees on a velocity for the estimation process and also on an estimation method and technique (e.g. playing poker for estimation, fibonacci series for story points).
The Business analyst explains the requirements to the rest of the team. The team votes on the story for determining the story point. Relative estimation based on reference sets is generally used to determine the size of the story prior to voting. Stories with large story points are decomposed further for estimating better and sized appropriately to fit in iteration. After the estimation process, based on the team velocity, the roadmap for the release is created. This is similar to the initial version of the project plan or work plan.

Figure 4 (a). Estimation tools- planning poker cards.
Figure 4(b). Estimation technique – planning poker game.

D. Project Planning (Sprint Planning)

The purpose of this meeting is to define the work plan and features that will be delivered as part of the sprint. The sprint or iteration is a time boxed event and may run between 2 weeks to 4 weeks as the organization deems appropriate. The business analyst works with the product owner to complete the user stories with details of the test and acceptance criteria.

The sprint planning meeting happens in the sprint zero for the initiated project and during the last week of iteration for a project in execution. The project manager works with the Scrum team to determine the availability of resources, their hours and the duration of the iteration, vendor availability for questions or files as applicable.

The Scrum team, along with the product owner, reviews the list of user stories identified for the current iteration or sprint. The story points allocated to the stories are also reviewed so that they accurately reflect the effort. The team velocity is validated against the list of stories & story points for that iteration. The team along with the product owner reviews the stories that have milestone dates or deadlines associated and mark them as high priority.

Figure 5(a). Sample user story card with story points and priority.
Figure 5(b). This figure displays the acceptance or done criteria for the user story along with the testing scenarios.

The team starts creating tasks that they need to perform for the iteration to complete all the user stories selected for the sprint. Separate tasks are created for testing, UAT, code move, implementation, file delivery, verification and confirmation tasks related to vendor, etc. The user stories along with the tasks are displayed on the Scrum board prior to the start of the next iteration.

E. Project Execution (Sprint)

The project development happens during the sprint. The sprint or iteration is a time boxed event and can extend from one week to multiple weeks for each iteration or sprint.

Daily scrum or stand up meetings during the iteration should last fifteen minutes to discuss the tasks the team members performed yesterday, their tasks for the current day, and any impediments that prevent them from performing their current day task. New user stories are added only if requested by the product owner and acknowledged by the team. Already prioritized stories for the current iteration can slide to the next iteration if the team and the product owner agree on that. The team members are not allowed to change the story points on stories and hours on the tasks during an iteration or sprint.

F. Monitoring Process (Sprint Review)

During the sprint review, the team provides a walk -through of all the items, features, and products that they have developed for that iteration. The product owner can test all the items from a regression testing standpoint and sign off for implementation or rollout. The business analyst can also explain if certain features are started in this iteration and will be completed in the next iteration. The technical team can also talk about the preparation that they did in this iteration for their tasks in the next iteration if need be.

G. Controlling Process (Sprint Retrospective)

The sprint retrospective is the main mechanism for taking the visibility that Scrum provides into areas of potential improvement, and turning it into results. It is highly recommended that the product owner be included in the first half of the meeting to get his/her feedback and then in the next half of the meeting, the team can discuss the details on what went well and identify areas for potential improvement. This meeting is usually facilitated by the Scrum master with established ground rules for decorum and discipline amongst team members.

III. DEFINING PROJECT MANAGER ROLE

An agile project relieves the traditional project managers from being task masters and being more of a leader who helps the team through difficult decisions about business value and priority. The primary role of the project manager is to focus on the product vision, motivate the team members, and foster solidarity and team spirit within the team. Project decisions which were previously managed by the project manager are now managed by the scrum team. In traditional projects, the project manager assumed the role of a problem solver, where as in the agile projects, this responsibility is now vested with the team. The project manager in agile projects is responsible for guiding the team and influencing them to be focused on the product vision. He needs to inspire and promote creative thinking thereby providing flexibility for the team innovation and autonomy.

IV. TEAM DYNAMICS

It is necessary for the organization to decide on how to balance the control and influence of project decisions distributed amongst the product owner and project manager within a project. There are cases where the product owners are controlling, and have a very high level of influence in the project execution right from selection of software, design of prototypes, design of tables, availability of data in certain storage locations etc. This limits the project managers from influencing or managing customer expectations while working with the Scrum team. The below figures illustrate two types of models which are quite prevalent in projects.

Figure 8(a). Represents a model where the Product owners are Controlling and exercise high level of influence.
Figure 8(b). Represents a model where more power is vested with the Project manager.

The involvement of the product owner is very critical to the success of the project, but their interaction and involvement must be limited to help with prioritization and provide clarity of requirements and product features and not project management. Agile projects promote the concept of sharing project responsibilities between the project manager and product owner.

Self organized teams are highly productive and performing teams. Any project methodology, be it traditional or agile, goes through the storming stage when the project team is assembled. In agile projects, the dysfunctional teams are exposed and any issues amongst team members are brought to light very quickly. It is the responsibility of the Scrum master and project manager to work in resolution of conflicts amongst the resources. Coaching and guiding through the Scrum process is recommended for teams starting to execute agile projects for the first time.

V. ESTIMATION TECHNIQUE FOR USER STORIES

The main reason for using the relative estimation techniques in agile projects is to move away from the traditional estimation using hours. Usually in traditional estimation, a buffer time is usually built in for handling uncertainty and this buffer time impacts the overall project time line. So many organizations prefer to use relative estimation instead of time estimates.

Relative estimation techniques are used for estimating the user stories. A reference set is established and the team can compare the user story to the reference set and provide a relative estimate for the story. During the sprint planning meeting, tasks are created for the user story and time estimates for completing the tasks are assigned.

It is also recommended to recalibrate the initial estimates at the end of every iteration to check if the relative estimates are accurate for the user story based on the task level time estimates as compared to the actual work performed. This recalibration exercise is usually performed after the sprint review and before the sprint retrospective to benchmark the team velocity accurately.

VI. GROOMING THE BACKLOG

Constant grooming of the product backlog is a key factor for the iterations to be successful. It is necessary to keep the user stories well groomed for a successful sprint planning session and subsequent sprints. It is mostly the key task of the business analyst to groom the user stories in consultation with the product owner on a regular basis. It is necessary for the business analyst to start looking ahead for the next iterations, to keep the stories well defined and gather the requirements for the next iteration in time before the sprint planning session. This not only helps the team, but also the project managers to identify risks ahead of time.

The Scrum team must frequently walk the board to discuss the stories and tasks on the current iteration and also to get a glimpse of how the next iteration is shaping up. It helps the technical team members to plan for any analysis tasks that they might have to do in the current iteration to be able start the actual development of the product planned for the next iteration.

The product owner also plays a critical part in the grooming activity, helping to prioritize and re-prioritize the stories in the roadmap for subsequent iterations. Grooming also facilitates the discussion for breaking down user stories into more manageable or well defined unit of work for estimating the tasks better. Grooming also helps to prevent new stories from being injected into the iteration in progress.

VII.INSPECT AND ADAPT

Agile projects can enter the realm of chaos if not monitored properly. In dynamic and non linear work environments, enthusiasm and creativity can work against the project if not controlled correctly. To paraphrase Thomas Jefferson, the price of agility on the edge of chaos is eternal vigilance.

The project manager must be the project watchdog to monitor the progress, enabling open communication among the team, to create an environment for learning, flexibility, and innovation.

In the sprint retrospective, the team members must be encouraged to discuss all the possible avenues for improvement without impacting the project delivery and timeline. Small iterations with feedback loops help to improve the process and increase productivity if managed effectively. Simple rules must be established to maintain team discipline and members must be encouraged to follow those.

VIII. CONCLUSION

Even though some organizations making a transition from traditional methodologies may view the agile methodology as being chaotic and lacking organization, they will see considerable cost savings and high-quality software products developed in shorter period of time as compared to traditional methodology.

It may be noted that "more than half of enterprises that aren’t already using agile processes are interested in adopting them." Forrester Research [11].

This transition is a balancing act. If the organization moves too quickly it can risk business continuity and stability. If it moves too slowly, it can put the company at a significant competitive disadvantage versus competitors who do take full advantage of agile capabilities. To lead the transition, IT leaders need to think about the long term organization goals, understand the current challenges, and not get trapped in the doom loop.

REFERENCES

1. Nandhakumar .J and Avison J (1999) The Fiction of methodological development: a field study of information systems development. Information technology and people 12(2):176-191

2. High smith, J.A. (2000) Adaptive software development: A collaborative approach to managing complex systems. New York N.Y Dorset House Publishing.

3. Introduction to Scrum for Project Managers. cPrime consulting.

4. Beck,K.,Beedle,M.,van Bennekum,A., Cockburn, A., Cunningham.,
   Fowler,M.,Greening,J.,Highsmith,J.,Hunt,A.,Jeffries,R.,Kern,J., Marick,B., Martin,R.,
   Mellor,S.,Schwaber,K.,Sutherland,J. and Thomas., (2001) Manifesto to Agile Software development (22.3.2002).

5. Agile Software development methods. Pekka, Abrahamsson. Outi, Salo, Jussi Ronkainen, Juhani Warsta.VTT publication 478.

6. Managing Agile projects. Edited by Kevin Aguanno. Multimedia publications (2005)

7. Economics of the Cloud. Microsoft November 2010.

8. Agile project management. CC pace Systems.

9. Happonen H., Hietanen V., Friman M., Harju T., Kiviniemi T.Results through Automation: Streamlining Operations and Maintenance Processes, Control Systems 2006 Conference proceedings CD-ROM, 6-8 June, Tampere, Finland.

10. Good to Great, Jim Collins 2001.

11. The Truth about Agile Processes by Carey Schwaber. Forrester Research.

12. Wheatley, Margaret. Leadership and the New Science: Discovering Order in a Chaotic World. Berrett-Koehler Publishers, 1999.

13. Cockburn, Alistair. Agile Software Development. Addison Wesley Longman, 2001.

Author : Narayan Parasuraman, PMP, CBAP, CSM, CSP, ITIL is a result oriented project manager with extensive experience in business analysis, project and program management. His core competencies include helping organizations with their strategic initiatives and managing process changes. He is also certified six sigma green belt and a scrum practitioner who focuses on helping organizations in their quest for agility. Narayan has led large-scale projects for big organizations and has a track record of managing and executing successful projects over the years in a variety of industry segments including financial services (retail banking, brokerage and portfolio management), insurance, supply chain management, logistics, manufacturing, and semiconductor industry. He has worked across the entire software development life-cycle and capable of delivering a wide variety of application architectures and brings to his consulting, passion and best practices gained by working with diverse and heterogeneous technical teams dispersed all across the globe. He is also very passionate about software development methodologies and has helped establish industry best practices in many organizations.