ANSWER

The following are the types of questions to ask when determining how to measure solution performance.  These will provide the business analyst guidance on how to support Solution Evaluation.

• What is the scope of the testing? What part of the solution is, and is not being, evaluated.

• Who will be measuring the solution performance?  Are the testers:
  • Day-to-day users who will use the solution extensively
  • High-level user that will use specific solution functions
  • Focused user groups, such as security, that will concentrate on certain solution functionality

• What is the expectation for the testing?  Do the testers want to conduct:
  • High-level, end-to-end testing (possibly evaluating an early prototype)
  • Detailed solution evaluation (evaluating whether specific requirements are met)
  • System performance testing (are performance metrics being met)

• What approach will be used to measure solution performance?  Will the testers:
  • Be given free range to “play with the system” (within the scope of the evaluation)
  • Be asked to follow scenarios a typical user would follow
  • Be instructed to validate specific functionality
  • Stress-test the system to see how it performs with a high number of users

• How will solution performance be measured?
  • Will outcomes be measured on a qualitative scale (Pass/ Fail) or quantitative (Rate on a scale 1 to 5)?
  • How will deficiencies be categorized (deficiencies can run from system error messages to editorial and formatting comments)?
  • Will deficiencies be grouped by functionality (is the deficiency found in a commonly used or a rarely used functionality)?

  --
Thomas Slahetka
LinkedIn Profile