The short answer
It depends on what you want to measure. If you want a percentage you can trust, such as "how many people would buy this again", you need a sample size calculated from your confidence level and margin of error. If you want to find out what is broken or confusing, a much smaller qualitative test is often enough.
Two kinds of product test
Quantitative tests answer "how many?" questions: share of testers who liked the scent, average rating for comfort, share who would recommend it. The result is a number, and the number is only useful if the sample is large enough.
Qualitative tests answer "why?" questions: what confused people when opening the pack, what they compared it with, what they would change. You are looking for patterns in what people say and show, not a precise percentage.
Many teams run a small qualitative round first to fix obvious problems, then a larger quantitative round to measure the improved version.
The sample size formula
For a quantitative question with a yes/no or percentage answer, the standard formula is:
n = Z² × p × (1 − p) / e²
- Z is the z-score for your confidence level: 1.645 for 90%, 1.96 for 95%, 2.576 for 99%.
- p is the expected proportion. If you have no idea, use 0.5, which gives the largest (safest) sample.
- e is the margin of error you accept, as a decimal. ±5% is 0.05.
A worked example
You want to know what share of your target customers would buy a new product again, with 95% confidence and a ±5% margin of error, and no prior estimate.
n = 1.96² × 0.5 × 0.5 / 0.05² = 3.8416 × 0.25 / 0.0025 = 384.16
Round up: you need 385 completed responses.
Adjusting for a small population
If the group you care about is small, for example 2,000 existing subscribers, apply the finite population correction:
n_adjusted = n / (1 + (n − 1) / N)
With N = 2,000: 384.16 / (1 + 383.16 / 2,000) ≈ 322.4, so you need 323 responses.
Plan for people who do not respond
The formula gives completed responses, not shipments. Not everyone who receives a product will answer. Divide the required responses by the response rate you expect from your own past campaigns to get the number of products to send, and run a small pilot first if you have no history to go on.
Margin of error in practice
Halving the margin of error roughly quadruples the sample. Going from ±5% to ±2.5% at 95% confidence takes you from 385 to 1,537 responses. Decide how precise the answer really needs to be before you commit the product budget.
Try it with your own numbers
Use the free sample size calculator to run these numbers for your own confidence level, margin of error and population. When you know how many testers you need, Portifer handles recruiting from your own lists or the UserLabs tester community, shipping, and collecting the feedback.