Beyond quality: how to ensure your BA efforts satisfy your customers?

It is a common question many managers ask themselves: “How to understand that my team is performing well?” Moreover, many skilled individuals do ask themselves the same thing: “How can I be sure, that I am doing well?” Especially in the business analysis sphere, the criteria of a quality job may be vague or only partly relevant.

In reality, what many people do is try to measure the quality of business analysis work through analyzing some aspects of BA artifacts they produce. Sometimes it is done in a generally acceptable way, e.g. when we speak about the results of requirements verification and validation as a means of measurement for BA quality. But sometimes it is done way worse, e.g. when we count changes to requirements after the base-lining or, even more inferior, assess quality by the amount of bugs in the final product that are traced to the requirements developed by a specific analyst. What makes me feel such attitudes even in the best-case scenario are only sub-optimal is that in fact they show the quality of completely different things. When talking about the requirements verification, we are assessing the quality of requirements, a single given artifact - not the business analysis as a separate service within the organization. The same goes with other metrics from this ballpark; what they really may measure is the performance of a practitioner.

Even if we find a satisfactory way to measure the performance of a BA (which is possible, of course), we should not think that this alone is a good assessment of the quality of undertaken BA activities. In this article, I'll try to explain why I believe that the performance evaluation on its own only partly covers the task of quality assessment and how we can ensure the expected level of quality for our own work or the work of the whole team.

When we speak about quality, we should understand the difference between the following terms: quality, efficiency, and effectiveness.

Efficiency - which is the thing most often measured intuitively - is a “relationship between the result achieved and the resources used”, in other words, a measure of profitability from efforts applied. It shows whether it is feasible to conduct business analysis in the company at all. In this respect, such metrics as the number of iterations or re-work on requirements may be suitable, even though not assessing the BA quality directly.

Effectiveness - which is rarely measured intuitively by the practitioners - is an “extent to which planned activities are realized, and planned results are achieved”. In other words, this measurement shows us whether the BA work brings any expected additional value to the business at all. Both definitions were taken from ISO 9000 [1].

An activity can be efficient but not effective, e.g. if a requirements document is created from the first attempt without any verification at all, it may be efficient (we use fewer resources) but is unlikely to be effective - such results may be useless or even counterproductive. And vice versa an activity can be effective but inefficient, e.g. when a simple work product is delivered by a team of ten analysts working full-time. They may bring a best in class artifact, but the marginal value of each analyst will be virtually infinitesimal.

In this way, a combination of effectiveness and efficiency as a measure of performance should be used when measuring the overall quality. Moreover, I believe that the quality of any process cannot be measured separately from the perception of it a customer has.

Here I call any beneficiary of BA activities as a customer. It may be a developer who works with the requirements, a stakeholder who has the needs to be addressed, a big boss who wants business outcomes, or a third party client for whom you conduct any kind of analysis, etc.

You may have a sound system of KPIs (a balanced scorecard, for example) and a lot of other formal metrics and verification checklists that show great results in terms of performance - but all of that is useless if your customer is not satisfied with your work in general; this means those KPIs do not match the expectations. The other extremum is when your customers are pleased with your results and deliver great products on top of your artifacts while you do not measure individual performance or conduct any preliminary checks at all. The first situation is unfortunate because the customer does not feel that their needs are addressed. However, the second situation is dangerous as well, as it is not manageable. I.e. the customer is happy now, but we cannot guarantee that we can keep them in such a condition for long enough.

Therefore, I would say that for business analysis the quality should be measured as an ability to continuously satisfy their clients with the business analysis artifacts they produce. What should be taken into account is that it includes not only the intended function and performance, but also a perceived value and benefit to the customer. It means that if your objective criteria show that the results are satisfactory, but your customer does not think so, you should admit having problems with quality. They may arise from issues in communications, in expectations, in planning or wherever - still, you cannot rate the quality of your job high in this case.

Figure 1 illustrates this concept.

Figure 1. The concept of quality

This definition is partly based on the ISO 9000 definition of the quality of a product or a service, see [1].

Being in the middle of a project how can we measure the quality as defined above? Surprisingly enough, the best way to do so is with the help of subjective measures, not objective ones like set KPIs. However, do not let the fact that the measures are subjective mislead you - this does not mean that they cannot be applied on a systematic basis.

What I recommend doing is to use a combination of continuously performed actions to monitor the current level of quality.

First, do not get rid of those performance measures you already have. Even though I do not advise that they be used as a sole way of assessing BA activities, they bring their value and can be easily used in combination with additional measures.

Then, try to collect feedback about your work. I suggest four different ways of doing so: 360 assessment, surveys, peer reviews, and 1-to-1 meetings. Let us have a look at them one by one.

360 assessment

A “360 assessment” is something that may be applied quite rarely, maybe once in a year. Usually, it may be used close to the annual individual performance assessment that many companies run before paying bonuses or promoting people. Frankly speaking, Wikipedia describes the method well enough [2]. I will just add that it is essential to select a proper sample of assessors - only those who really have worked closely enough with this or that analyst may give a full and reliable opinion on their work. Additionally, this should be a really 360 assessment, i.e. people from different organizational roles should participate. It is good to start the request with an honest description of why you ask the person to participate. I may recommend the following pattern:

"Dear Mike, I'd like to give as objective an assessment as possible to Jane's work this year. I know that she has closely worked with you as a business analyst on the following projects: <...>. Could you please share your opinion on how well she performed and what can be done to make her work better?"

You may always add specific questions if you need to, especially having experienced any thorny issues during the projects. If you can ensure the full anonymity of such feedback it is even better, but in case you cannot, just promise the assessor that the answer will not be shared with anybody.

The same technique may be used to assess not only individuals but also teams as a whole.

Milestone surveys

A “Milestone survey” is a technique that you may use after you complete some major milestone or an iteration. For example, when the project involved organizing training sessions for the stakeholders, I have always conducted a post-training survey for the participants to measure the value from the training. Moreover, especially for the educational events it is a good idea to repeat the study after some time to see whether the participants experienced any residual value over time. Such surveys are better organized in the form of questionnaires and be relatively short.

Peer reviews

Peer reviews may be used as a source of internal assessment of a sampled piece of a work product. Such reviews may cover requirements documents, business process maps, meeting agenda or minutes, plans etc. When you have several analysts in your team, it is likely that they have different experience and may bring valuable advice on the artifacts they see. Such reviews may be both a means of measurement for the current quality level and a source of opportunities for improvement. You may refer to BABOK for more details about different types of reviews [3].

1-to-1 meetings

A “1-to-1 meetings with the customers” technique may be called the most essential one among these, but it is helpful only when applied regularly. From my perspective, this is one of the most underestimated means of both controlling the current quality level and ensuring that it will keep high in the future. It is a good idea to establish regular meetings with the principal customers to talk face-by-face about any problems occurred or any misunderstandings spotted. However, if the things go just as planned, such meetings should not be skipped, as they are a nice way to confirm that the customer is still on the same page with you.

It is a common situation when someone's own feelings about the work progress or intermediate results differ significantly from what others think. So if you are working on a requirements package, don't miss an opportunity to talk to the people who will be impacted by them once ready. Plan some regular meetings with you stakeholders from business, SMEs as well as with your developers and testers. Remember that the sooner any potential problem is identified the cheaper it is to deal with it.

All of the measures stated above are more likely to be reactive (maybe with the only exception of 1-to-1 meetings that can be used both as a means of assessing the work already done and ensuring that the future results will meet the expectations). As a car cannot be controlled only by looking at rear-view mirrors, quality cannot be provided basing only on reactive actions. To succeed one has to be proactive.

To do so, we need to understand the expectations of our customers and moreover to be able to manage them. A kick-off meeting or proper project management documentation can provide with the initial expectations of the parties, but they tend to change faster than the project documents usually do. Regular 1-to-1 meeting will help you stay tuned with the current expectations; this is an active action from you as a BA. At the same time, you may give your customers ability to act proactively too.

Here comes an idea of transparency. Indeed, the more transparent the activities are, the easier it is to meet the expectations. When the customers have all the information they need, they may signal if the things start to go wrong. On the other hand high transparency calls for high costs of maintaining it. How to keep the balance? A good instrument is a transparency-trust matrix (see figure 2).

I first learned about this matrix during one of Alexander Orlov's [4] webinars on managing relations with stakeholders. The idea is that any initiative (providing it is the first of a kind) starts in the sector A of the matrix - both transparency and trust are pretty low. Increasing the transparency (moving to B) we eventually raise the level of trust the customer has in us (moving to C). Finally, the customer keeps trusting us even without the transparency level they demanded previously (sector D). Unfortunately, should any issues occur, the level of trust starts decreasing. I would like to point your attention to the red line across the diagram - everything to the right of it is a comfort zone where the customer is more or less pleased with the control they have. Everything to the left is the danger zone, where the customer is likely to be dissatisfied. The rule of thumb is “when we lose in trust (or face problems), we increase in transparency”. This will help us keep the customers informed enough thus satisfied while staying as efficient in terms of resources consumed for reporting as possible.

Figure 2. Transparency-Trust matrix

Of course, this approximate model cannot be strictly calculated. Still it gives a good understanding of the whole attitude to transparency.

Another instrument here is defining the forms of control beforehand. If the reporting forms, the timings, and channels for reports are defined and followed the situation is more predictable.

The theory of process management tells us that for a process to reach its intended result a set of enablers need to be considered [5]. So lacking focus on any of them increases the risk that our process becomes inefficient or even ineffective. These enablers are:

Process Design

You need to have an established business analysis workflow. This does not mean that you need to document every single step and variation in the BA activities for your organization. But the main steps and values plus instructions for the most crucial parts of the process should be designed beforehand. For example, documents templates, the order of sign-off, the rules for using IT systems, the responsibilities of the participants and so on - you'd better have all of these in place by the time you or your team needs it.

IT and infrastructure facilities

Be sure that your software is set for the BA tasks you are going to perform with it. E.g. all the requirements attributes can be captured in the system, the modelling notation is included on the package, and the access rights are set. At the same time, it is crucial to check the working environment - e.g. the level of noise, the light, the air-conditioning. All these factors may affect productivity.

Personnel

It is obvious that only a worker sufficiently trained for the task may complete it on their own. Still, sometimes we are short of resources, knowledge, motivation, etc. All these aspects should be addressed to ensure the high productivity of an employee.

Policies / Context

Whenever starting a new task, it is advisable to check for existing policies, laws, or standards in the given area. This enabler may be not major for the BA field, but still your organization may have some special rules for the domain you are going to work in - it is better to understand them.

Understanding the context reveals unobvious factors that may affect your work.

Measurement
We have already talked a little bit about the types of control above. The general thing is that all stakeholders should understand the control in place. What are the control points, why are they present, what can the consequences of any violations be. Five types of control may be differentiated:

• final (control only by results)
• preliminary (control of the pre-final results to leave some space for changes before the finale)
• periodical (control based on time periods)
• per milestone (control based on reaching defined milestones)
• sampled (a randomized form of control).

Be sure that all the stakeholders (and yourself) understand which type of control is used and why.

If you are a team manager, you can monitor all these elements constantly to help your team deliver great results. As an individual, you may check the enablers before any single project/task to reveal potential risks you may face.

Concluding, if we agree to measure the quality as an ability to continuously satisfy the clients with business analysis artifacts, then to keep it controllable we need to perform a set of quality checks such as 360 assessment, 1-to-1 meetings, surveys, etc. and to keep it predictable we need to take into account the five enablers stated above.

Paraphrasing the contents of ISO 9001, to ensure a planned result you need to know what is expected, find a skilled worker, provide them with a suitable instruction and working environment, set a measurable goal, and implement relevant monitoring [6]. This is a minimum necessary set to keep the consumers of your work products satisfied.

Author: Igor Arkhipov, CBAP.

Igor holds Master of Business Informatics degree specializing in Business Process Modeling and Optimization. Igor has broad experience as a Business Analyst and a BA Team Manager in the fields of customer support&services, software development and e-commerce. https://au.linkedin.com/in/igarkhipov

1. ISO 9000:2015 Quality management systems – Fundamentals and vocabulary, Fourth Edition / ISO (2015)
2. https://en.wikipedia.org/wiki/360-degree_feedback (date of reference 07-03-2016)
3. A Guide to the business analysis body of knowledge (BABOK), v.3 / IIBA (2015) ISBN-10: 1927584027
4. https://www.linkedin.com/in/amorlov (date of reference 07-03-2016)
5. Workflow modelling: Tools for Process Improvement and Application Development, 2^nd Edition / Alec Sharp, Patrick McDermott (2008) ISBN-10: 1596931922
6. ISO 9001:2015 Quality management systems – Requirements, Fifth Edition / ISO (2015)