“Write a set of use cases to implement two-factor security in a mobile app.”
Use a prompt like this with a large language model (LLM) like ChatGPT from OpenAI, and you’re likely to be impressed with the output. Not only is the content generated almost instantly, the completeness and quality of the documentation won’t differ much from what an experienced business analyst would produce after hours of hard work.
Recently I started to notice an increase in messages from business analysts asking for my feedback for project artifacts created using ChatGPT. LLM-based tools are being used to generate not only system requirements, but also meeting agendas, lists of questions to ask in stakeholder interviews, change management plans, and more. And the conclusion is clear: even though LLMs remain prone to “hallucinations” (generating false statements), they now can perform many of the tasks typically assigned to business analysts, as long as human oversight exists to instruct the model, validate outputs, and tailor the content to specific business contexts,
This quote from Seth Godin summarizes well the impact that generative AI is having on the business analysis work:
We’ll never again need to hire someone to write a pretty good press release, a pretty good medical report or a pretty good investor deck. Those are instant, free and the base level of mediocre. The opportunity going forward remains the same: Bringing insight and guts to interesting problems.
The era in which the bulk of the activities performed by business analysts consisted of creating pretty good specifications and pretty good acceptance criteria is now over.
I avoid making predictions, but my bet is that in 2024 we’ll start to see fewer and fewer BA job ads focused on documentation tasks.
As I write this article in February 2024, many job boards are still advertising business analyst positions with a heavy emphasis on documentation. Examples of activities listed include the following:
- Gather customer requirements, and turn these into well-defined product requirements and specifications.
- Elicit and document business requirements and document business processes.
- Create and maintain use cases, process flows in various diagrams such as activity diagrams, UML diagrams, sequence diagrams, etc.
- Collaborate with stakeholders to understand business requirements and translate them into functional specifications.
Two decades ago, I developed a successful career in business analysis relying primarily on my strong requirements documentation skills. While I never considered it to be my most valuable competency, this is what consulting customers highlighted every time they invited me to take another project or referred me to other clients. Soon companies started to ask me to train their analysts, and for ten years I taught an online course called Crafting Better Requirements.
Still, as more professionals start using LLM-based tools to produce grammatically and technically correct requirements documents, less companies remain dependent on trained analysts to write project documentation. Most of the techniques I developed over the years to help BAs avoid common requirements flaws are now irrelevant, because with the right prompts ChatGPT can successfully prevent or correct the majority of these issues.
And this is far from bad news! We should celebrate the fact that leveraging an LLM tool, business analysts can finally shift their effort away from documentation tasks and concentrate on high-value activities that solve important problems or move opportunities forward.
We all know what those activities look like: working across organizational boundaries to build bridges, understanding the problem to be solved, identifying valid alternatives, recommending optimal solutions, drawing insights to find improvement opportunities, using critical thinking and analysis to troubleshoot issues and improve processes, and so forth.
To understand why anyone with core BA skills beyond documentation should not be worried about being replaced by AI, consider a recent test I performed using ChatGPT:
“An executive asked for a dashboard to monitor KPIs. What would be good questions to ask so the best solution for the executive’s problem is found?”
In response to this prompt, the chatbot suggested a series of questions, all revolving around implementing the dashboard. For instance:
- What are the specific business goals and objectives you want to monitor with this dashboard?
- “Who are the intended users of the dashboard?”
Any skilled business analyst should be able to immediately detect the problem here. Just because a stakeholder asked for a dashboard, it doesn’t mean that’s the best solution for their problem. At this point, we haven’t even defined what the business challenge is!
Something that we understand and LLMs don’t is that we humans are hardwired to lead with solutioning. We get a dopamine high for finding the solution to a problem even before the problem has been properly defined. And because LLMs are trained on human-generated content, naturally they fail to perform the kind of reasoning that only a knowledgeable BA would be able to apply to avoid the dangers of solution-based thinking.
Unlike ChatGPT, a skilled analyst wouldn’t jump to the conclusion that a dashboard is the best alternative to address the business need. They’d ask questions to clarify what progress the executive is trying to make; what would be different if they had a dashboard; what they’ll be able to do then that they can’t do now. During this discovery process, they might learn that what the executive is truly after is business insights to help improve decision making. Only after the business problem had been clearly defined the BA would start looking into viable alternatives to solve it. Here, in addition to a dashboard, the desired outcome might be enabled by a business report, or a system that notifies business leaders when important KPIs are trending up or down.
Someone simply following ChatGPT’s instructions, on the other hand, would be rushing into solution mode and risk missing a better choice. Now, does that mean that LLM-based tools can’t be used to improve and expedite BA results beyond improving documentation? Not really. Ideally, the current generation of generative AI models would be better at things like avoiding premature solutioning, effectively correcting users on misguided queries, and qualifying their responses with appropriate levels of confidence. But even with the current flaws, LLM tools can dramatically accelerate our progress if we develop a solid understanding of their capabilities and limitations.
Making ChatGPT work for, not against us
LLMs can have a positive impact on the results we deliver. To go back to our example, after investigating the business problem and fully understanding the business challenge, we could ask ChatGPT for alternatives to a dashboard to produce business insights. When I performed this experiment, I got a long list of options. While some of the suggestions had to be discarded because they’d only apply to a limited business context, others helped expand the initial set of valid alternatives under consideration. Now the business could move forward with a more robust list of alternatives for the solution evaluation phase.
Other useful applications of LLM-based tools include:
- Leveraging Copilot.ai to generate code to query internal databases to validate a business hypothesis, rather than waiting for a data engineer to be available to perform the task for you.
- Asking ChatGPT to act as “devil’s advocate” and produce evidence or standpoints that differ from your argument to help you avoid blindspots.
- Using Google Bard to create a summary of a complex technical topic easily understandable by non-technical people.
Of course, you should never forget to check your organization’s internal policies for allowed use of LLMs. While some companies already have agreements and security measures in place to create a secure environment for using generative AI to enhance employees’ productivity and efficiency, others are still banning or restricting its use to prevent issues like accidental leaks of private, confidential, or proprietary information. But if you follow the policies governing acceptable use and apply common sense (e.g., never upload internal source code or sensitive meeting notes to an unauthorized application), LLM-based tools can help you maximize the value you deliver to your organization.
How not to be replaced by ChatGPT
Companies that treat the documentation of customer stated needs, wants, demands, desires, ideas, specifications as the focal point of the BA work tend to pay a steep price by solving the wrong problem or addressing a problem manifestation rather than the underlying business issue. And now, with LLMs readily available to produce similar outputs at a much faster pace, there is reason to believe that these document-centric roles will soon join the statistics of LLMs replacing human jobs.
An adept BA, on the other hand, has no reason to fear an existential threat. It’s hard to predict exactly how generative AI will end up integrated into our work lives going forward. Still, LLMs are far from being able to send stakeholders in the desired direction, identify the initiatives that actually make sense and are likely to produce the desired result, and reliably answer the question, How does this project/feature/requirement we’re working on contribute to the organization’s strategy?
It may take a while for job postings to adapt to the new developments, especially when so few organizations remember to update their job description boilerplate language on a regular basis. But regardless of what current job ads say, the ability to write high-quality requirements documents is rapidly becoming an obsolete skill.
Luckily for us, our capacity to combine deep thinking and analysis with compassion and empathy will remain a rare and sought-after competency. And if we keep using AI tools as a booster to make our strengths even stronger, we’ll continue to be unbeatable at “bringing insight and guts to interesting problems.”
Author: Adriana Beal
Adriana Beal spent the past two decades helping innovation companies leverage decision science and machine learning to improve business outcomes. She recently left her job as a principal data scientist with a global AI consulting group to return to her roots as an independent consultant. You can find out more about her work visiting bealproject.com.