How usable and satisfying is my finished product and how does it stack up against the competition?

How it works.

A comparatively large number of users – about 25 – complete tasks using the finished product. These tasks represent the critical success factors against which the product will be benchmarked. Critical metrics are captured from each session, e.g. ease of use, satisfaction, tasks success rates, and time required to complete tasks. A scorecard is produced which shows how well the current version of the product measures up against earlier versions of the same product or against competitive products that have been previously benchmarked using the same method.

Pros:

• Summative tests provide a clean consistent means of tracking progress toward User Experience goals.

Cons:

• Since summative tests usually include time-on-task measures, participants are not encouraged to “think out loud” during task completion. This leads to less richness in qualitative data compared to Formative Testing. Summative tests take the longest to conduct because of their large sample sizes.