Before an organization releases a new piece of software or web feature to all of its customers or the general public, it will generally offer a limited audience a chance to test drive the feature and offer their feedback. This is generally known as a Beta launch, though some companies have other terms for it such as early adopter feedback or limited user-trial.
By the time an organization reaches the Beta launch stage, they have, hopefully, discerned that their product that meets the project’s basic needs by employing numerous visualization tools. Use case diagrams, prototypes, and even more sophisticated software applications such as iRise or Axure enable an analyst to present something very close to the end product to stakeholders and even sample users fairly early in the development cycle, prior to time investments of designers and engineers.
So by the time an organization gets to a Beta launch, an organization is just a step away from releasing the product or web page to the masses. By exposing the product to a controlled group of responsive users, a Beta should reveal any bugs or surprises in the software, thereby helping an organization control risks. The Beta stage can be incredibly revealing or they can be an exercise in futility, depending on how they (and the feedback they elicit) are managed. Often falling under the business analyst’s and/or project manager’s purview, a good Beta launch entails clear, diligent communication between analysts, engineers, project manager, business owners, and the Beta audience. Use the following tips to help you use your next Beta launch to your company’s maximum advantage.
(Note: Some companies’ sites or software act in a constant state of Beta, with the company continually tweaking bits here and there to test whether their users like it. Yahoo is a very notable example of this. If the change works for users, it stays; if not, it comes down quickly. This article deals with the more traditional notion of Beta as a feature of early product testing.)
The Planning Stage
-
Time your launch to be most convenient to your users. In academia, for example, do not launch your Beta before a summer or holiday break. For European users, do not launch a Beta during the summer months when many users will be on very long (weeks at a time) vacations. If you are leery about releasing too early, note that the product does not need to be completely bug-free upon launch, or even have the engineers believe it to be so. According to author Scott Belsky, “If there are fundamental flaws in your assumptions about your product, you will realize them more quickly if it’s live. Rather than spending many months (and lots of money) on the finer details, getting early feedback can lead to priceless realizations.”1 In other words, you’ve got to identify and address any big problems before you can fine-tune the details, and a beta launch is the best way to do that. If you are pressed for time, consider launching completed portions of the Beta as they come available. This will keep the Beta in front of your audience as you contact them notifying them of each new portion and then follow up on the last portion, creating a sense of pace and urgency associated with the launch and their participation in it.
-
Determine how many users you need to get good Beta feedback. Here, it’s almost always a better idea to overbook than under book since some of users may not have as much time as they think they will to experiment with the product.
-
Secure your audience. Choose experienced customers and newbies (perhaps in consultation with your sales or training staff). Also choose national and international customers (if applicable), exposing any unknown cultural or technological barriers inherent in the product. If disabled users will be in your audience, try to recruit representatives from that group as well. Determine, in concert with the business owners and engineers, how long the soft launch will last so that the Beta audience can be advised. Too long (months) and you risk losing users’ interest. Too short (days) and you risk lack of participation. (Prior to inviting your audience to participate or providing any log-in credentials, advise them of any nondisclosure agreements.)
The Feedback Stage
-
Invite your Beta audience to use the product or tool or to view the page in their daily work. Provide an overview of what the product is supposed to do, along with help copy, if applicable. But think twice about providing a video tutorial or anything that a regular user would not have during the actual launch. At this point, your research with visualization tools and the like should have ensured that the product is intuitive for users.
-
Monitor your users’ usage via their IP addresses. They should be actively participating. If after a week someone hasn’t tried the product, send tactful reminders of how essential their input is. Invite any members of your sales force or training staff with whom the users have a relationship to follow up, also.
-
When feedback does start to flow in through ad hoc questions or comments, expect criticism. Betas are not perfect, nor should you expect them to be. According to an article on DotNeil.com from a writer who’s witnessed many private Beta launches, “As soon as outsiders start using the application, bugs will crop up - literally, straight away. Some will be complete show-stoppers.”2 So during this stage, adjust your (and your team’s) expectations.
-
In addition to inviting users to send their comments ad hoc via email, instant messaging, and so on, you may want to implement a formal way for users to provide feedback to ensure that all user concerns are addressed—and thought of. (While ad hoc comments may bring up some issues your team never would have thought of, conversely formal questioning may elicit concerns that your users would not have thought to voice.) So follow up with a questionnaire, virtual focus group, or both.
Virtual Focus Group
-
If your user group is very proficient and very limited (i.e., 10 or less), you may be able to hold a virtual focus group inviting verbal feedback. But use this approach with caution. You must ensure that all of your participants feel their views have been heard and are not annoyed when they get off the phone or go offline, feeling that they tested the product for nothing. Use the analyst focus group skills you’ve honed in your requirements gathering to ensure that your group does not stray off topic or engage in group think or gold plating.
Questionnaire
-
Tips on designing questionnaires abound, but remember that your goal here is to gather as much data as possible from your limited user group, so ask exploratory, open-ended questions that invite lots of comments. Qualitative questions (such those on a Likert scale) will not reveal as much or be as helpful.
-
Keep it short—six questions or less (the longer the questionnaire, the less likely users are to complete it).
-
Keep it general, i.e., What about the design did not work for you? What about the function did not work for you? What would you change? What would you keep?
-
Give users the opportunity to express what you might not have asked about. Leave at least open-ended, catch-all essay question; i.e., “What did we not cover that you would like to change about this product?”
The Analysis Stage
-
Sort user responses into general categories and groups (layout, functionality, quality of data, accessibility, and so on). Send them to your team, and then hold a meeting to select and prioritize fixes prior to the actual launch.
-
Advise your Beta audience of fixes that are coming. If any of their suggested changes will not be forthcoming, tactfully explain that their feedback is valuable, but their suggestions may require longer term planning and evaluation.
-
Follow up to cement relationships for future beta launch feedback. Don’t underestimate the value of thank you gifts (leather portfolios, edible goods, etc.) and notes. Assure users of how invaluable their input has been, and let them know that they have helped shape the product for their industry
Remember that meticulous planning can ensure that business analysts and business owners get the most information out of these launches, helping to garner a smoother, more successful actual product release. Select the points above that work best for your organization and work style, and enjoy your planning and Beta!
Author: Morgan Masters is Business Analyst and Staff Writer at ModernAnalyst.com, the premier community and resource portal for business analysts. Business analysis resources such as articles, blogs, templates, forums, books, along with a thriving business analyst community can be found at http://www.ModernAnalyst.com
1“Belsky, Scott. “The Beta Principle: Skip Perfection and Launch Early.” Accessed 20 September, 2010.
2“Five Tips for Your Private Beta,” Accessed 20 September 2010.