User stories can describe what an AI agent should accomplish. They rarely define how much power it should have, when it must stop, or who is accountable when it makes the wrong decision.
Consider the following user story:
As a customer service manager, I want an AI agent to resolve routine customer complaints so that customers receive faster service.
It sounds reasonable. It identifies a user, a capability, and a business benefit. A delivery team could estimate it, place it in a backlog, and begin discussing acceptance criteria.
It is also dangerously incomplete.
Can the agent issue refunds? If so, how much? Can it access the customer’s payment history? Can it make exceptions to company policy? Can it send a legally significant response? What should it do when customer records conflict? When must it obtain human approval? How will the organization reconstruct what happened if a customer challenges the outcome?
The user story does not tell us.
That was less of a problem when software primarily waited for a person to click a button, enter a value, or approve a transaction. AI agents can do more than present information. Depending on their design and permissions, they can interpret a situation, select among possible actions, use external tools, communicate with customers, update records, initiate transactions, and continue working through a multistep process with limited supervision.
For these systems, describing the desired outcome is not enough. We must also define the boundaries of delegated authority.
AI agents still need user stories, use cases, process models, business rules, data requirements, and acceptance criteria. But they also need an additional requirements artifact—one designed specifically to answer a more consequential question:
What is this agent authorized to decide and do on the organization’s behalf?
That artifact is an AI Decision Contract.
User Stories Were Designed for Software That Waits
Traditional business applications generally operate within tightly defined workflows.
A user selects an account. The system retrieves information. The user chooses an action. The system validates the request. A person approves the transaction. The application records the result.
Even when such a system contains extensive automation, its decision paths are usually established in advance through deterministic rules:
- If the transaction exceeds a threshold, route it for approval.
- If a required field is missing, display an error.
- If the customer meets all eligibility conditions, enable the next step.
- If a policy exception is requested, send it to a manager.
The user remains visibly involved, and the software generally performs an action because a person directly requested it or because a predefined rule clearly required it.
An AI agent can operate differently.
An agent may receive a goal rather than a complete sequence of instructions. It may determine which information to retrieve, which tool to use, how to interpret ambiguous input, what intermediate steps to take, and whether the task has been completed. Modern agent designs may combine language models with instructions, tools, organizational data, memory, and control logic so they can perform multistep work rather than merely answer questions.
That flexibility is valuable, but it changes the requirements problem.
When a conventional application calculates a refund from explicitly coded rules, analysts can inspect the inputs, formula, and result. When an AI agent interprets a complaint, reads policy documents, examines account history, evaluates exceptions, chooses a response, and initiates a refund, several forms of judgment have been delegated to the system.
The requirements must therefore define more than functionality. They must define authority.
The Dangerous User Story
Imagine that a project team receives this requirement:
As a claims manager, I want an AI agent to evaluate and settle straightforward claims so that customers receive faster decisions.
The team develops an agent that can:
- retrieve the customer’s policy;
- examine supporting documents;
- estimate the covered amount;
The agent fulfills the user story. It evaluates claims and settles them quickly.
But what does “straightforward” mean?
Is a claim still straightforward when two documents disagree? What if the requested amount is unusually high? What if the applicable policy language is ambiguous? What if fraud indicators are present? What if the claimant is in a jurisdiction with special notification requirements? What if the agent is uncertain but still selects the most probable interpretation?
A team could technically satisfy the user story while building a system the business would never knowingly authorize.
This is not merely a hypothetical concern about imperfect AI output. Security and risk-management guidance increasingly emphasizes the danger of giving AI systems excessive autonomy or permitting high-impact actions without sufficient oversight, validation, and limits. OWASP refers to this class of risk as excessive agency, while NIST’s AI Risk Management Framework emphasizes defined responsibilities, human oversight, monitoring, documentation, and the management of risks throughout the AI lifecycle.
The problem begins before development. It begins when the requirement describes the objective but leaves decision authority undefined.
What Is an AI Decision Contract?
An AI Decision Contract is a proposed business-analysis artifact that specifies the authority, constraints, evidence, escalation rules, and accountability surrounding an AI-enabled decision or action.
The word contract does not necessarily mean a legal contract. It means an explicit agreement among business owners, risk partners, product teams, technology teams, and other stakeholders about what authority is being delegated to the agent.
The Decision Contract establishes the operating boundaries within which the agent is expected to work.
A useful contract should answer questions such as:
- What decision is the agent being asked to make?
- What actions may it take independently?
- What actions are prohibited?
- What information may it use?
- What financial or operational limits apply?
- When is human approval required?
- What evidence must support the decision?
- What information must be retained for review?
- How can an incorrect action be reversed?
- Which human role remains accountable?
A Decision Contract does not replace a user story.
The user story explains the stakeholder need and intended value. The Decision Contract explains the boundaries of delegated judgment and action.
Together, they provide a more complete requirement.
The Nine Parts of an AI Decision Contract
1. Decision Objective
First, define the business decision or outcome—not merely the activity the agent performs.
Weak objective:
Process customer complaints.
Stronger objective:
Determine whether a customer complaint qualifies for a refund under the current refund policy and, when authorized, initiate the appropriate resolution.
The stronger version identifies the actual decision: whether the customer qualifies and what resolution is permitted.
The objective should also connect to a measurable business outcome, such as reducing resolution time, improving consistency, lowering handling costs, or increasing customer satisfaction. Speed alone should not be treated as success if the agent produces more incorrect, unfair, or unrecoverable outcomes.
2. Permitted Decisions and Actions
Specify what the agent may decide or execute without additional approval.
For example:
- determine whether the purchase falls within the standard return period;
- classify the complaint using approved categories;
- offer a replacement for an eligible product;
- issue a refund up to a defined amount;
- send a response using an approved communication template;
- create a case and route it to the appropriate team.
Avoid vague statements such as “handle normal cases.” Terms such as normal, routine, simple, and low risk must be translated into observable conditions.
3. Prohibited Decisions and Actions
Some boundaries should be absolute.
For example, the agent must not:
- change a customer’s contractual terms;
- waive a regulatory requirement;
- make an adverse decision based on a protected characteristic;
- disclose restricted personal information;
- approve a transaction involving an employee’s own account;
- make payments above its delegated limit;
- delete evidence used in reaching a decision;
- represent uncertainty as confirmed fact.
Prohibited actions deserve explicit documentation because teams otherwise tend to focus on what the system should do rather than what it must never do.
4. Data Boundaries
An agent should not automatically receive access to every available data source.
Define:
- restricted or prohibited data;
- data-quality expectations;
- permissible uses of personal information;
- whether the agent may infer new information from existing data;
- what it should do when sources conflict.
For example, an agent evaluating a refund may need the transaction record, product information, refund policy, shipping status, and prior case history. It may not need access to unrelated customer data.
Limiting access is both a business requirement and a security control. An agent with unnecessary data or tool access creates additional opportunities for misuse, disclosure, or unintended action.
5. Authority Limits
Delegated authority should be measurable.
Examples include:
- a maximum contractual commitment;
- a maximum number of affected accounts;
- a limited set of eligible products;
- approved communication channels;
- geographic or jurisdictional limits;
- restrictions on irreversible actions;
- a maximum number of automated attempts.
A useful principle is:
The greater the potential impact, the narrower the agent’s independent authority should be.
A $15 courtesy credit and a $15,000 claim payment should not have identical approval requirements merely because both are technically “refunds.”
6. Confidence and Evidence Requirements
A numerical confidence score may be useful in some implementations, but it should not be treated as a universal or perfectly calibrated measure of correctness.
Instead of relying only on a model-generated confidence value, define the observable evidence required for the decision.
For example, an agent may issue a refund only when:
- the transaction is verified;
- the request falls within the stated return period;
- the refund policy contains an applicable rule;
- no fraud or account-restriction indicator is present;
- the customer’s identity has been sufficiently verified; and
- the relevant records do not conflict.
The requirement can also identify situations in which the agent should acknowledge uncertainty rather than forcing a conclusion.
The goal is not to create the illusion that every decision can be reduced to a precise probability. The goal is to establish the minimum evidence that must exist before the agent is permitted to act.
7. Escalation and Stop Conditions
An AI agent must know not only how to proceed, but when not to proceed.
Human review may be required when:
- required evidence is missing;
- authoritative sources conflict;
- the requested amount exceeds the agent’s limit;
- a policy exception is requested;
- the decision may create legal or regulatory consequences;
- the customer disputes a previous automated decision;
- fraud or abuse indicators are present;
- the agent encounters a situation outside its approved scope;
- the proposed action is difficult or impossible to reverse;
- the system cannot verify that a required control was completed.
Escalation should be treated as a successful outcome when the case exceeds the agent’s authority. An agent that correctly stops is behaving better than one that produces a confident but unauthorized answer.
NIST guidance recognizes that human-AI configurations can range from fully autonomous operation to systems that defer decisions to human experts. The appropriate arrangement depends on the context, impact, and risks of the use case.
8. Explanation, Logging, and Traceability
When an agent makes or supports a significant decision, the organization should be able to reconstruct what happened.
Depending on the use case, the record may include:
- the policy or rule version used;
- the data sources accessed;
- the recommended or completed action;
- the responsible business owner.
This does not mean preserving unrestricted internal model reasoning or exposing hidden technical processes. It means retaining enough business-relevant evidence to understand, review, audit, and challenge the decision.
The explanation shown to a customer may differ from the operational record retained by the organization. Both should be defined.
9. Reversal and Accountability
Finally, define what happens when the agent is wrong.
Can the action be reversed automatically? Who investigates the issue? How is the customer restored to the position they were in before the error? Does the agent lose authority after a threshold of exceptions? How are similar completed decisions identified and reviewed?
Most importantly, identify the accountable human role.
The accountable person may not review every decision, but the organization should not allow accountability to disappear merely because an AI agent performed the work.
An agent can execute a decision. It cannot assume organizational responsibility for authorizing the policy, setting the limits, accepting the risk, or correcting systemic harm.
From User Story to Decision Contract
Consider this original user story:
As a customer support manager, I want an AI agent to issue refunds for eligible complaints so that customers receive faster resolutions.
Here is a corresponding Decision Contract:
The agent may issue a refund of up to $75 when the purchase is verified, the product is covered by the current refund policy, the request falls within the standard return period, no fraud or account-restriction indicator is present, and the applicable policy rule can be identified.
The agent must not issue a refund when customer records conflict, eligibility depends on a policy exception, the customer has submitted multiple related claims within 90 days, the requested amount exceeds $75, or the proposed resolution would affect contractual rights.
Cases outside these boundaries must be routed to a customer support specialist. Each automated decision must record the transaction, policy version, evidence used, action taken, customer communication, and responsible business owner. Refunds must remain traceable to the original case and must support an approved reversal process.
The user story communicates the business need.
The Decision Contract communicates the limits.
Now the delivery team can design the agent, its permissions, its workflow, its user experience, its integrations, its tests, and its monitoring around an explicit delegation of authority.
Five Acceptance-Criteria Categories AI Stories Commonly Miss
Traditional functional acceptance criteria often focus on whether the system produces the expected result for a given input.
AI-agent requirements need additional categories.
Authority
Can the agent perform the action, or is it merely allowed to recommend it?
Given that a requested refund exceeds $75,
when the agent determines that the customer is otherwise eligible,
then it must route the case for human approval
and must not initiate the refund.
Uncertainty
What happens when the available evidence does not support a reliable conclusion?
Given that the purchase date differs across two authoritative records,
when the agent evaluates refund eligibility,
then it must identify the conflict,
avoid making an eligibility decision,
and escalate the case for review.
Explainability
Can the result be explained in business terms?
Given that the agent denies an automated refund request,
when it communicates the decision,
then it must identify the applicable policy condition
and provide the customer with an approved review or appeal path.
Recoverability
Can an incorrect action be corrected?
Given that an authorized reviewer reverses an automated decision,
when the reversal is completed,
then all affected systems must reflect the corrected outcome,
the original decision must remain traceable,
and the customer must receive an approved notification.
Observability
Can the organization detect poor performance or unexpected behavior?
Given that the agent is operating in production,
when its outcomes are monitored,
then authorized users must be able to review escalation rates,
override rates, exception patterns, disputed decisions,
processing times, and outcomes by relevant customer and case segments.
These criteria move the conversation beyond “Did the agent complete the task?” to “Did it operate within its authority, handle uncertainty appropriately, and leave the organization able to review and correct the outcome?”
The Business Analyst’s Emerging Role
The rise of AI agents does not reduce the need for business analysis. It increases the need for careful analysis of decisions, policies, risks, and accountability.
Developers and architects determine how an agent can interact with models, tools, APIs, databases, workflows, and security controls.
Business Analysts help determine:
- what business outcome the agent should pursue;
- which decisions are suitable for automation;
- what information is relevant and permissible;
- whose interests are affected;
- what business rules and policies apply;
- where human judgment remains necessary;
- what exceptions must be anticipated;
- what evidence must be retained;
- how success and harm will be measured;
- who owns the resulting decision.
IIBA has similarly described business analysis as increasingly important to AI governance, decision intelligence, strategic alignment, human oversight, and accountability.
A useful way to describe the distinction is:
Developers determine how an agent can act. Business Analysts help determine when it should act, when it must stop, and who answers for the consequences.
This work cannot be completed by the Business Analyst alone. Legal, compliance, risk, cybersecurity, privacy, operations, product management, data, architecture, and affected business stakeholders may all need to contribute.
But the Business Analyst is well positioned to bring those perspectives together and translate them into a coherent, testable agreement.
A Practical AI Decision Contract Template
The following template can be adapted to the risk and complexity of the use case.
AI DECISION CONTRACT
Decision or action:
Business objective:
Business owner:
Affected stakeholders:
1. SCOPE
The agent is intended to:
The agent is not intended to:
2. PERMITTED AUTHORITY
The agent MAY decide:
The agent MAY execute:
3. PROHIBITED AUTHORITY
The agent MUST NOT decide:
The agent MUST NOT execute:
4. DATA BOUNDARIES
Permitted data sources:
Restricted or prohibited data:
Data-quality and freshness requirements:
Rules for conflicting or missing data:
5. AUTHORITY LIMITS
Financial limit:
Operational limit:
Geographic or jurisdictional limit:
Time limit:
Volume limit:
Restrictions on irreversible actions:
6. REQUIRED EVIDENCE
Evidence required before acting:
Applicable policies or business rules:
Identity or authorization checks:
Required validations:
7. HUMAN OVERSIGHT
Human approval is required when:
The agent must stop when:
Escalation recipient:
Required escalation information:
Maximum escalation response time:
8. COMMUNICATION
Information shown to the affected person:
Explanation requirements:
Approved communication channels:
Review, appeal, or dispute path:
9. LOGGING AND TRACEABILITY
Information that must be recorded:
Policy and model version information:
Tool and data access records:
Retention period:
Authorized reviewers:
10. REVERSAL AND REMEDIATION
How an action can be reversed:
How affected parties are restored:
How related decisions are identified:
Incident and problem-management process:
11. MONITORING
Accuracy or quality measures:
Escalation rate:
Override and reversal rate:
Complaint or dispute rate:
Bias or outcome-disparity measures:
Performance review frequency:
Conditions that suspend automation:
12. ACCOUNTABILITY
Accountable business role:
Technology owner:
Risk or control owner:
Final approval authority:
Not every low-risk use case needs a lengthy document. A meeting-scheduling agent and an agent authorized to move money should not receive the same level of analysis.
The artifact should be proportionate to the impact of the decision.
However, every agent that can take meaningful action should have an explicit answer to the same fundamental questions:
What may it do?
Under what conditions?
Using which information?
Within what limits?
When must it stop?
Who remains accountable?
Do Not Eliminate the User Story—Complete It
“Stop writing user stories for AI agents” is intentionally provocative.
The more precise recommendation is this:
Stop treating the user story as a complete requirement for an AI agent.
User stories remain useful for communicating needs and organizing delivery. Use cases remain useful for describing interactions. Process models remain useful for showing workflows. Business rules remain useful for defining policy. Data models remain useful for clarifying meaning and structure. Nonfunctional requirements remain essential for security, privacy, reliability, performance, accessibility, and maintainability.
The Decision Contract complements these artifacts by addressing the distinctive risk created when software is authorized to exercise judgment or take action with limited supervision.
This distinction also prevents a common mistake: attempting to solve every AI risk with better prompting.
A prompt can instruct an agent to behave cautiously, but an enterprise control cannot depend solely on the hope that the agent will interpret a sentence as intended. Authority should also be enforced through permissions, workflow controls, tool design, validation, human approvals, transaction limits, monitoring, and technical guardrails.
The Decision Contract defines the business agreement. The system design must enforce it.
The Question Every AI Project Should Ask
Take the most important AI-agent requirement in your backlog and ask:
Could the system technically satisfy this requirement while doing something the organization would deeply regret?
Could it complete the task but use inappropriate data?
Could it reach the desired outcome but exceed its authority?
Could it satisfy the user story but make an unfair decision?
Could it act correctly in most cases while failing dangerously in rare ones?
Could it produce an answer without enough evidence?
Could it take an irreversible action when it should have asked for help?
If the answer to any of these questions is yes, the requirement is not yet complete.
The next generation of business analysis will not be defined only by how well we describe what systems should do.
It will also be defined by how clearly we specify what intelligent systems are allowed to decide, what they are forbidden to do, when they must defer to a person, and who remains responsible after automation has acted.
An AI agent needs more than a user story.
It needs a contract for its decisions.
Author: Johnathan Mitchell, Enterprise Business Analyst
Johnathan Mitchell is a seasoned Enterprise Business Analyst with over 20 years of experience steering large-scale transformations and complex projects to success. With a keen eye for detail and a strategic mindset, he specializes in bridging the gap between business objectives and technological solutions. Johnathan has worked with Fortune 500 companies on projects across various industries, including finance, healthcare, and technology, helping them navigate the intricacies of enterprise-level initiatives.
His expertise lies in requirements engineering, process optimization, and stakeholder management. Johnathan is known for his ability to dissect complex business problems and devise innovative solutions that drive efficiency and growth.