<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=204513679968251&amp;ev=PageView&amp;noscript=1">

Do You Even Data

A data-driven marketing blog

Want to learn how you can translate incredible data list information into killer marketing campaigns? Want to better understand how data research and models can enhance the data you already have?
All Posts

Fundamentals of Product Testing

Here’s our approach to product testing.

  • Monadic experimental design: If you have 5 new concepts, each respondent in the study should be asked to evaluate only one of them. There should also be a control concept…typically the current product and/or a competitor’s version.

When you show a respondent more than one concept, the comparisons become a beauty contest and not a science. Going to market on which product people like better is a mistake. The degree to which something is liked is a good metric (also known as hedonic scaling), but that’s only one of the elements that we suggest you measure. Further, by showing the respondent a single concept, you get more precise information about how people are reacting to that specific concept (and can compare this to your control) without introducing any bias or noise associated with subjective comparisons to a sometimes-confusing array of choices.   Individual concept performance against the control is a very reliable, predictable analysis in a monadic environment.

  • Measure purchase intent with caution: this is pretty much always overstated by respondents.

There are some steps we can take to mitigate this overstatement of purchase intent through a modeling algorithm based on multivariate regression.  You can have some refined purchase intent information, comprising the key metrics used in a concept test, to come up with a predictive composite value.  The highest composite score relative to the control wins.  Better yet, stated purchase intent can often be calibrated to actual historical sales data or other action standards, enabling more predictable absolute numbers instead of relative ones.

  • Measure the attributes that drive trial: when a consumer tries something for the first time, it’s not an accident.

Take into account high believability of product claims and its uniqueness. To establish a basis of comparison, use competitor concepts or the existing product on the market as a control.  Pick the design that has the best chance of first-purchase success and, at the same time, alienates the fewest current customers who currently buy the existing product.

  • Measure the appropriate product attributes: use statements couched in a Likert-scaled phrase like “On a scale of 1 to 7, how much do you agree with the following statements…”

The statements should be articulated directly toward the product (e.g. freshness, whether family wants to eat it, brand I trust, good value for the money, etc.).  The proper statements will help differentiate between the concepts and will ultimately drive your marketing message.

  • Compare the data: report individual pieces of data for comparison.

Your analysis will be often be less about the absolute value of the scores of the products than it is the deltas (differences) between the scores.  Don’t be surprised when the existing product (your control)—which has been in consumers’ consciousness for a much longer period of time than the 10 minutes of this survey—wins; this is very common in concept testing.  We’re looking for the best new concept performance which will approximate that of the control and, importantly, outperform competing products.


Dino Fire
Dino Fire
Dino Fire

Dino serves as President, Market Research & Data Science. Dino seeks the answers to questions and predictions of consumer behavior. Previously, Dino served as Chief Science Officer at FGI Research and Analytics. He is our version of Curious George; constantly seeking a different perspective on a business opportunity — new product design, needs-based segmentation. If you can write an algorithm for it, Dino will become engaged. Dino spent almost a decade at Arbitron/Nielsen in his formative years. Dino holds a BA from Kent State and an MS from Northwestern. Dino seems to have a passion for all numeric expressions.