Whether you’ve never heard of Agile or you just finished your nth Agile project, you need to understand that Agile is here to stay! Are you, the Business Analyst, an extinct species in this new world? Is your career changing? Do you need new skills?
Agile guru and visionary Scott Ambler talked with Adrian Marchis, ModernAnalyst.com's Publishing Editor, and shared his vision on what’s next for Agile and his thoughts on the role of the business analyst in the Agile world.
Modern Analyst.com: First and foremost: What is Agile?
Scott Ambler: There are so many important features of agile:
- It’s evolutionary in nature so it’s iterative and incremental.
- Agile approaches are highly collaborative so we try to communicate as practically as possible.
- We try to avoid documentation and prefer more effective means of communication such as face-to-face.
- We’re self organizing, so that the people that do the work are the ones that are best suited to organize and plan it. That’s a bit different.
- We do some documentation. We will hold some reviews but we’re really smart about it, and we focus on value.
- Other important thing is the focus on quality. We are producing higher quality than we did in the traditional world. This is due to the focus on iterative approaches and factoring, testing and stuff like that.
- We work closely with stakeholders resulting in higher stakeholder satisfaction.
- And probably one of the more radical things, one of the more radical aspects of agile is that we welcome changing requirements. A change in requirement late in the lifecycle is a competitive advantage as long as you can act on it.
So, this is a bit different for a lot of traditionalists. We rethought things a bit and, as a result, we have higher levels of productivity and higher levels of success.
ModernAnalyst.com: One thing you mentioned that is common in agile is the focus on collaboration. Does agile work for projects that have distributed teams?
Scott Ambler: Yes it does actually. This is one of the things that we looked into in the Agile Adoption Survey.
We specifically asked about that and we explored the success rate in three different scenarios:
- co-located,
- near located, where you’re in the same building and you could get together on a daily basis if you had to; that might mean you need to drive but you could do it if you had to.
- Then far located, it’s all plane to get there physically.
We found that co-located agile team had an 83% success rate. Near location team have a 72% success rate and far located had a 60% success rate. So, there is a risk premium the more distributed you become and the implication being that you want to throw better process at those approaches and better tooling. The real implication is that you don’t want to be distributed if you can avoid it.
And if you have to be distributed (if you have people working from home or on different floors in the same building), organizing an approach like that is significantly different than organizing a co-located one. So, this is something the project management community, really needs to pay attention to it because the risks are measurable and have been measured.
So, it shouldn’t be news to anybody. People are doing distributed agile.
ModernAnalyst.com: Another thing you mentioned is that agile is a very iterative process and that iterations are at a core of agile. So how does agile differ from RUP (Rational Unified Process)? Isn’t RUP also an iterative process?
Scott Ambler: RUP is iterative. RUP actually introduced and socialized a lot of the concepts that the agile community takes for granted like the focus on delivering working software on a regular basis, on iterations, on collaborations, on working iteratively, on delivering software incrementally, on testing throughout the lifecycle. So, it really sort of paved the way for the agile community. And RUP can be as agile as you want to make it.
There’re wonderful studies out there, case studies and books and stuff like that on how RUP can be agile. I was writing about how RUP could be agile long before I joined IBM. So, I think that’s an important thing to note. There’s a lot of knowledge out there on agile approaches to RUP.
And what I’m also seeing is that as the agile community is figuring out how to scale and do more complex things; that they are adopting a lot of the techniques that have been in RUP for a long time. It’s frustrating to watch them reinvent the wheel because they apparently don’t want to read about RUP. They like to beat up on RUP a lot. They beat up on RUP a lot but not realizing that there’re a lot of really great ideas … that they can take advantage of. But we’re an industry that loves to reinvent the wheel I guess, so I guess we have to do that yet again.
ModernAnalyst.com: How did you arrive at agile? Have you always worked on agile projects or did you start with more traditional approaches?
Scott Ambler: I’ve been all over the board so it’s interesting. I guess years ago, I was very agile not knowing it. Then I guess in the mid 90s, as I’ve gone to bigger and bigger projects, I started working in a more formal manner and got heavily into CMMI (CMM at the time) for a few years. But really sort of saw the formal approaches struggle. And beginning in the late 90s, began moving away from it. That’s actually one of the reasons why I became so interested in RUP… I was kind of looking at Scrum at the time, too. I was very heavily into process for almost two decades now, I guess. So, I was always looking for better ideas of how to work.
So, yeah, I’ve been on both sides of the fence I guess and can safely say that agile does appear to work. It does work better in practice than the more formal traditional stuff. And a lot of the stuff around traditional element isn’t really formal at all. It certainly isn’t disciplined. It’s more bureaucratic and I think there’s a lot of bureaucracy that gets touted as formalism and touted as discipline.
ModernAnalyst.com: So, you’re saying if I just use a template, it’s not necessarily disciplined, right?
Scott Ambler: Yeah, exactly. There’s actually a really great book out right now, that I don want to suggest to your readers… The book is called “Adrenaline Junkies and Template Zombies.” It’s a collection of patterns and anti-patterns for IT. So, one of the anti-patterns is template zombies; those are people who just fill in templates and making sure it’s filled out. Whether or not it makes sense, just fill it out. Whether or not it’s even the right template, but I have a template, I’m going to fill it out. It’s just a novel waste of time. A lot of wastage in general…. It’s a really interesting book and one of the really neat things about it … they present these patterns and anti-patterns just as a collection of one or two pages in length. … I suspect it will end up being one of those instant classics that we’ll be referring to 20 years from now.
ModernAnalyst.com: I noticed a couple of times you’ve mentioned something that a lot of the people in the traditional world would be surprised; and maybe even people from the agile world. Your survey respondents indicated that they actually used documentation?
Scott Ambler: Yeah. We ran this very detailed survey on how people are modeling and documenting in general. It wasn’t an agile specific thing. So, what we did was we started out with the usual, “Who are you” type questions. We also asked to choose a type of project that you’re going to describe throughout the rest of the survey. They’re given the choices of agile and traditional and iterative and ad hoc. The really neat thing is that it split out almost evenly. So, roughly a quarter, 25% plus or minus five we had in each of these four categories.
It was really interesting to see the habit when it comes to modeling and documentation. One of the things that the agile community is harping on for a few years now, is that agile is actually more disciplined than traditional. I have run the numbers exactly but just eyeballing things, it appears that many of the modeling and documentation practices you see on traditional projects, you also see on ad hoc projects.
One of the reasons why I run these surveys is I want to find out what’s actually happening because there’s a lot of religious discussion out there… And I use the term religion on purpose because people just have these religious beliefs over what works and what they should be doing. What is the proper way of working? There is no data to back it up, right? Particularly in a traditional community, there’s a phenomenal amount of theory out there, which has never really been backed up or if it was looked into, the numbers are basically show: oh-oh, that was a really bad idea to begin with. So, I’m trying to gather the numbers to find out what’s going on.
What was one thing that we found was that the agile folks are just as likely if not more likely than traditional people to write deliverable documentation like user manual, operations documentations, support documentation, system overview documentation. The true deliverable type stuff, which I thought was a bit surprising. So, I was thinking about this a bit; and what I think was going on is one of the things that we do know, is that Agilists are actually more likely to deliver than traditionalists. That type of documentation is something that you would write towards the end of the project as things stabilize. So, I think what’s going on is Agilists are more likely to get to the end, therefore, they are more likely to do the type of work that you would do at the end, which is finish up this documentation. That’s my theory at least. I can’t back that up with real numbers but that’s what the trend appears to…
I was surprised that Agilists were just as likely if not more likely to write documentation.
That’s not what I expected.
ModernAnalyst.com: Switching tracks a bit, is there a need for business analysis, the discipline, on the agile projects?
Scott Ambler: Yeah. There is a need to do analysis. For anything even remotely interesting, yeah, you’re going to be doing analysis. But it’s not a phase; doing analysis in a phase is actually a phenomenally risky way of working. It might not be a specific job role. So, I think this is where the business analysis community [needs to pay attention] and this is true of any community; is true of testing, true of project management, true of many other things, true of programming. You really don’t want somebody just focused on that. You need to do the activity but that doesn’t mean you need a specialist in it.
ModernAnalyst.com: You don’t think there is a place for a business analyst role on an agile project?
Scott Ambler: There’s a lot of great opportunity for business analysts. There’s definitely a great opportunity for people with business knowledge and skill on an agile project.
For example, we’ve got this role of an onsite customer or product owner; somebody who represents the business to the team. That’s a role that most business analysts, particularly those that are coming from the business side, are almost ideally fitted for. Because if you look at that product owner role, that onsite customer role; they do a lot of analysis activities:
- They’ve got to negotiate between a wide range of people.
- They have to communicate the requirements to the development team.
- They have to represent what’s going with the development team to the overall business community.
So, there’s a lot of good stuff there for business analysts.
But it’s not a strict business analysis role. We need more than what a business analyst can typical do. They need to prioritize the requirements, because we work in priority order, which enables us to produce higher return on investment. We need people that can come on who can make decisions right now; even if it is not perfect decision, give us your best answer now, but let us know that it’s not the best one…
We don’t need a big document. We don’t need to have somebody to go back to the Steering Committee and wait for the official meeting two weeks from now to make a relatively trivial decision. We need the decisions now…
The analysis effort is very important but it is done in a different and more effective manner than what we see in the traditional world. For the business analyst who wants to step up in that role: Hey great! Do it! …
I’m not saying any of this is easy but it’s a very important role and very interesting. So, I think there’s a lot of opportunity there.
But, if your vision is to write detailed use cases or write detailed data models or whatever your models are of choice and then hand it over to the development team; that’s not of much value. That’s actually increasing risk. We don’t need that.
Assuming people are flexible, and I know a role for a business analyst is just to become a team a team member because everybody on the team should have analysis skills. This is something that’s very easy for people to gain if they’re given the opportunity. So, in the agile world, one of the things we try to do is just grow our skill set, have a wider range of skills and pair with people to work closely with them and collaborate with them. That that way we can learn new skills from them and they can learn new skills from us. So, this actually spreads information amongst the team faster, lowering cost and risk and also increases our skills faster, which also reduces cost and risk as well. So, this is a good thing. But people need to be flexible. Anybody coming in with business analysis skills will start picking up other skills from other team members. They might pick up testing skills, programming skills, whatever. Not that they’re going to become an expert programmer or an expert tester, but they’ll pick up some of these skills and will become better at these sort of things. This is actually critical and this is a big difference.
ModernAnalyst.com: So the analyst needs to do more than just business analysis…
Scott Ambler: One of the challenges I think in the traditional community is that they’ve rewarded people for becoming overly specialized. You’ve got people who are just programmers, who are just testers, just business analysts, just project managers. In the agile community, we’re going away from that because that’s seen as a pretty big risk.
We’re moving more towards craft people, which is probably not an accurate term but I like to use the term ‘generalized specialist’, somebody who has one or more specialties, because you better be able to do something useful, but who’s got a general knowledge of the process and good knowledge of the domain as well.
So, that’s in between. On the extremes, we have generalists and specialists. A generalizing specialist is the person that’s somewhere in between those two extremes and the ‘best of both worlds’. This is the type of paradigm that we are starting to see in the agile community.
ModernAnalyst.com: So you’re saying that business analysis is not as much of a role or phase, within the agile world, but rather a discipline and set of skills?
Scott Ambler: Yeah and it’s pretty more like a discipline or a set of skills that one activity gets. A nice way to say it is that business analysis is so important to agile people that we do it every day. It’s not a phase. Actually when I see an analysis phase in a project that’s telling you that analysis must be a phenomenally trivial thing that we only have to do it one point in the project and then for some reason we don’t ever need to do it again or very rarely need to do it and desperate try to avoid it through the change management process.
Analysis is so important to us we do it on a daily basis. We want to be working close with our business stakeholders. We want to be working on a daily basis to get information from them, to get feedback from them. A lot of that interaction with the business stakeholders is obviously going to be some sort of analysis type of activity.
As a result, we can’t really afford to have only one person that does analysis on the project or small group of people doing it because they very quickly become bottlenecks. We need people who are willing to do analysis and can do it on a moment’s notice. If I realize I need to talk to somebody and get some information out of them, I should go straight to them. I shouldn’t have to go to Sally the business analyst and wait for her to be available or schedule a meeting with her. Then have her schedule a meeting with the person I actually need to talk to and all that stuff. That’s like a bridge or a go-betweener or a Band-Aid. It’s not an effective way of working. It’s better than not doing analysis but it’s definitely not the best way of working.
ModernAnalyst.com: Are there any specific analysis skills or competencies that you see are of big value on the agile project?
Scott Ambler: It’s all the important ones that I would argue that you see in the traditional world:
- The ability to communicate with people, to ask questions, and get information out of them.
- To negotiate priorities between different groups - because the business stakeholders never have one single voice on their own. They need somebody to help facilitate that, somebody who knows the business to begin with.
- The business domain knowledge, period, is important. It is something that everybody needs to have, not just business analysts.
What’s not as critical, is the specification. Yeah, obviously in a regulatory environment that’s a little bit different…
One of the things in the agile world is that instead of documenting a requirement, we prefer to capture it in the form of a test. So, if you believe that a requirement can be testable, then the next implication is why don’t you just skip over the documentation middleman to begin with and just start out by writing the test. Just cut out the extra busy work. As long as that test is human readable and then you can tie it in with the actual working code; hey, you are off to the races, right? So, just a slight change, just adopting the practice of executable specification, you can have a dramatic impact on your productivity. But it requires a different skill set on the part of whoever is doing that analysis specification effort.
ModernAnalyst.com: To complement those competencies and skills, what tools and techniques do you see the analyst using on an agile project?
Scott Ambler: The techniques are all there, they’ve been there for a long time. A lot of the techniques in agile modeling are actually reworking those user centered design concepts from the late 80s, early 90s.
So, doing a little bit of modeling at the beginning, initial requirement positioning… just spend a couple days, maybe a week or so of gathering initial requirements to do basic scoping so somebody can ask you for some sort of rough estimate or rough schedule. Some sort of description of what you’re going to build.
That doesn’t mean that you need to create detailed documentation with it. It means you need to do a little modeling. Modeling on a daily basis. Do model storming. Doing iteration modeling at the beginning of each iteration…
As far as tooling goes, the most common tools are white boards and paper, particularly for working with stakeholders; this is a uniquely specific tool technique to allow them to be actively involved in the modeling effort.
The people are also using drawing tools. The more sophisticated ones are using CASE tools. At Rational, we got a very interesting product called Rational Requirements Composer. It’s a more simplified modeling tool that’s geared for the business analyst that enables them to do analyst level modeling to get the value out of modeling, which helps to communicate but not take the hit of unnecessary documentation.
If there’s a need for detailed documentation then RequisitePro or DOORS; those are better options for you. But for the vast majority of projects that aren’t under a regulatory environment, something like Requirements Composer would probably be better off for them.
ModernAnalyst.com: So, on an agile project, there isn’t much need for the more traditional requirement management tools such as RequisitePro or DOORS?
Scott Ambler: Yeah. All things being equal, it’s not a paradigm issue, it’s an environment issue. If there’s a regulatory need for strict requirements traceability or strict requirement specification, then you’re going to have that need if you’re agile or traditional. Those are the type of factors that are going to drive the decision for do you need a more formal requirement management tool; at least in my mind.
ModernAnalyst.com: Are you suggesting that, for the vast majority of the projects, agile can be the methodology of choice?
Scott Ambler: I would think so, yeah. When you look at the facts, all the numbers point toward agile… and away from traditional. But the problem is that the religion that we see around process and the IT community doesn’t see them; so ends up not motivating people to question their beliefs.
There’re a lot of people that really still believe in the traditional way of working and don’t really understand agile, don’t want to understand the implications of traditional and yet they still cling to their old beliefs. It’s really unfortunate.
So yeah, all things being equal, the vast majority of projects should in fact be agile. And we’re seeing a movement towards this… But it requires greater discipline. It does require people have a wider range of skills, to be more open minded, to be flexible. So, some of the things that we rewarded people for in the traditional world sort of hamper them in the agile world.
Some people are not going to make the jump to agile and not everybody needs to. There will still be traditional work out there but not so much over time.
ModernAnalyst.com: What advice would you have for business analysts who are pondering moving on to agile project or have landed a job on an agile project and they’re wondering, “Now what?”
Scott Ambler: I think the important thing is to be flexible; to view documentation as almost always as a risk. You need to do some documentation but it should be your option of last resort. You should always be asking, “How can I achieve these goals in a more effective manner?”
The real goal of analysis should be: let’s identify the requirements, let’s explore and understand what people actually need, and let’s communicate and make sure that the developers actually do that…
- You’re going to have to be actively involved in the team because there’s not going to be eight hours a day of analysis work to do. Splitting yourself across multiple teams is highly frowned upon in the agile world. That should be highly frowned upon in the traditional world as well. We need to be able to go and talk to somebody with business knowledge right away.
- Becoming focused on capturing requirements specifications in the form of tests is critical. You’re still going to be doing some diagramming and high level stuff; you’re never going to get away from that. But the detailed requirement specifications, if you’re not capturing that as test, that could be a problem. This is a very clear direction that we’re seeing in the agile community. So, I highly suggest doing that.
- Being prepared to work with others; actually trying to pick up skills from other people, trying to help people pick up your analysis skills. Wanting to collaborate is vital.
ModernAnalyst.com: Is there a case or a place for so called “live system documentation”, documentation that evolves over time and is kept up to date for various reasons?
Scott Ambler: Yeah. That’s why in the agile community, there’s this focus on executable specifications, because then a specification actually adds true value to the developers, which in this case just validates their work and to do so in an test manner, they’ll keep it up to date, right? Because, if I change my code and I break a test, I’ve either got to fix my code to make sure it still performs that test or I’ve got to update that test. And if that means to renegotiate the requirement, then so be it.
Now can everything get specified in the form of tests? No. But can the vast majority of information be captured that way? And this, I think, is one of the reasons why we see Agilists producing measurably higher quality and achieving measurably higher business stakeholder satisfaction. We’re doing more testing and more testing generally leads to higher quality.
And working iteratively, working closer with stakeholders increases the chance that you actually understand what their true needs are and then implement that and act accordingly. This is why working in priority order helps insure that we get things out the door faster because we are much more likely to code the critical functionality first.
And then showing visible progress in the form of working software, in each iteration, is absolutely critical. It gives the business stakeholders visibility into the project. This is one of the very interesting implications of agile is that because we are working in a more collaborative manner, because we’re working in a more open manner, our stakeholders know exactly what’s going on. They can see software being produced regularly. They know exactly what they’re getting for their money. They can actually steer the project and assure they get software that meets their needs and not just something that’s built to spec. Building something to spec is a phenomenally risky way of working.
ModernAnalyst.com: Anything else that you want to get out there to business analysts about the role of the business analyst on agile projects?
Scott Ambler: I think the important thing is be open-minded about agile… If you find yourself in this position where you think Agilists don’t do ‘X’. Please go on to Google or your search engine of choice and Google the term “Agile X” and you’ll find there’s a wealth of material out there. I constantly run into people that tell me Agilists don’t document. They don’t model. They don’t test. They don’t do this, they don’t do that. You’re not doing yourself a favor by having a gross misunderstanding.
And the other thing is to just recognize that agile is coming your way. If it’s not in here now, it soon will be. It’s very clearly growing. It’s just not a fad. We’re on the exact same sort of adoption crew that would sell all this technology in the 90s. So it’s here to stay. All the numbers show that this works. This works in practice…
Don’t look for excuses not to do agile or not to be agile. There’s always an opportunity to be more agile no matter what you’re doing. agile is a continuum. So, you might not be doing everything that we’re talking about but if you can adopt one or two agile ideas to improve the way that you work, then maybe on the next project adopt one or two more ideas.
Take time to invest in yourself, invest in your career. Read and talk to people. Find out what’s actually going on.
ModernAnalyst.com: What about certifications?
Scott Ambler: Don’t trust some of the certification efforts underway right now. They’re okay.
But, for example, an interesting thing that we sort of seen get tweaked out of some of the surveys is that, the project management community seems to have a different culture and measurably different set of beliefs than everybody else, including business stakeholders. I’ve attributed that to the certification that we see from the PMI and the Prince 2 folks; the answers that the project management community gives on the surveys are much more closely aligned with what they’re being certified at, which shouldn’t be a surprise. But that is not the value of everybody else.
Arguably the book of knowledge, these books of knowledge and these certification efforts are…. There is a lot of good stuff there, don’t get me wrong, but some is not so good. And we need to start distinguishing between the religion and the reality. I think people are being still sort of certified in religion in some ways.
ModernAnalyst.com: Are you saying the problem with these bodies of knowledge is that they seem to imply that these are the only tools and that they work for all cases?
Scott Ambler: Yeah. The fundamental challenge is that if you look at the project management community, a lot of their book of knowledge was actually written based on theory.
Some of it is also there is some generic project management stuff that doesn’t apply to the IT world but people are still being certified in it. And that’s a fair thing but it’s a bit of a challenge. But there’s also some interesting theory that’s being taught that really doesn’t apply. It’s not what people want to hear. It’s not what the theory guys want to hear but they haven't taken the time to really look into this. They sort of know there’s a problem but they haven't figured out the solution. So, people are being certified in that. This is a challenge.
So, what do you do?
If you’re a professional and you’re being told to certify in these things for these good reasons, you’re going to learn that stuff. And because you’ve gone through all this effort and you’ve taken the courses and read the books and written the tests, the brainwashing has been accomplished. Now you’ve got to fight off all that.
You’ve more than likely surrounded yourself with other certified people or other people who want to be certified. So, you have this group thing stuff going on as well… There is the underlying assumption that what you’re being certified in is actually what you need to be certified in. I wouldn’t want to make that assumption.
There are some good reasons to do certification but you need to go beyond that. There needs to be more in your professional life than just certification.
ModernAnalyst.com: Do you have the same views on agile certifications?
Scott Ambler: … there really aren’t many coherent, respectable certification in the agile community.
ModernAnalyst.com: Are you also saying that there shouldn’t be?
Scott Ambler: There’s value in certification. But you need to be certifying people in something that is of value, something that’s real, and something that works. And not what we want to work because that’s what we’ve been told should work by theory people and by the textbook people.
ModernAnalyst.com: The IIBA is currently working on revising the Business Analysis Body of Knowledge (BABOK). What advice would you have for them?
Scott Ambler: I am one of the reviewers so I’m obviously now biased. They’re working on it. There’re challenges. Right now, we are at this unfortunate point in time where we’re doing this major transition from one process paradigm to another.
I think BABOK is forcing a little more than it should on the traditional stuff right now. A lot of people are doing great work and their hearts are in the right place. Their timing could be better because they’re just at this unfortunate point in time where things really haven't settled down yet on what really is best practice in the analysis world…
In the agile community, we’re not exactly known for putting stakes in the ground and trying of certify ourselves. The agile analysis certification movement is pretty much nil. The agile community might not be stepping up on this topic as much as we should.
ModernAnalyst.com: What are you up to these days? What are some of the fun exciting things that you’re working on and what are some trends that you see coming up?
Scott Ambler: Most of my stuff I’m focused on is helping organizations become more effective at software development in general. I’m only dealing with the more complex environments. My focus is on:
- Scale… scaling agile.
- How do you do agile at the enterprise level?
- How do you do this across teams of hundreds of people within regulatory environments?
These greater complexities that often get overlooked by the agile religious bigots among us. ;-) …
One of the things I’ve been saying for a while now is that when you look at what’s happening in the industry, the agile community has figured out now how to build high quality small applications. When you compare that to the traditional community, I would argue that they’ve figured out how to develop low quality scale applications. I’ll let the Legacy systems out there speak for that.
We might not see it in our lifetime, but the next round is that we really have to figure out how to develop high quality enterprise scale applications. I don’t think we’ve figured it out. I’ve definitely got some ideas in the form of the Enterprise Unified Process, and Agile Modeling, and Agile Data, and Agile Unified Process. But in general, we just haven't figured it out.
The traditional community clearly has not figured it out. There’s too much rampant bureaucracy and the governance efforts almost always a questionable effort at best amongst these organizations. There’s a lot of good will or a lot of good ideas but, in practice, it hasn’t worked for us. I think it will take at least a few more decades to figure this out if we ever do.
I think that’s going to be the next round.
That’s pretty interesting to me.
ModernAnalyst.com: Thank you for your insights!
[This interview occurred on August 1st, 2008.]
More resources:
About Scott Ambler:
Scott Ambler is the Practice Leader Agile Development within the IBM Methods group. He is an award-winning author of several books focused on the Unified Process, Agile software development, the Unified Modeling Language, and CMM-based development. He is a regular speaker at international IT conferences and is a contributing editor with Dr. Dobb’s Journal. He holds a bachelor's degree in computer science and a master's in information science from the University of Toronto. For more information on Scott, visit: http://www.ambysoft.com.