Quantitative questions for interviews… 8 Steps to prioritize customer needs

Why use quantitative questions for interviews, not just qualitative? Two reasons. First, qualitative interviews give you reams of customer quotes with no good way to prioritize customer needs. Sure, you could count the number of times customers cite a certain need, but frequency of remarks is a poor substitute for customer eagerness to improve.

More important, with only qualitative interviews you’ll succumb to confirmation bias… the tendency to hear and interpret information according to your preconceived notions. With quantitative questions for interviews, you’ll have hard data on customer needs that your team hasn’t filtered or skewed. You get 100% unadulterated insight straight from the customer.

You get 100% unadulterated insight straight from the customer.

We’ve been coaching B2B companies—mostly large multi-nationals—in the front-end of innovation since 2005, and seldom see companies consistently using quantitative questions for interviews. That’s a shame, because the results are so powerful. Our research shows these interviews “greatly or significantly” impact new product designs for 84% of teams.

It’s also a shame because it’s simply not that difficult to employ quantitative questions for interviews. Just follow these 8 steps.

Step 1. Target a single market segment and JTBD

A market segment is defined as a “cluster of customers with similar needs.” If you try to develop a single new product for multiple market segments, your new product won’t satisfy any customers to the fullest extent possible. By definition, different market segments have different needs, right?

If your company makes colorants, your target market segment might by paint producers. But your project scope is still too broad: You need to target a specific job-to-be-done (JTBD) by those paint producers. Their job might be, “production and sale of semi-gloss paints.”

Companies serving B2B markets can only improve their customers’ products and/or processes. Producers of materials and components have a big impact on customers’ products if their new offering is integrated into their customers’ products. Producers of equipment and services often have their greatest impact on their customers’ processes. Your quantitative questions for interviews will help you sort this out.

Step 2. Uncover all customer outcomes

Before you ask quantitative questions for interviews, you should ask qualitative ones. Why? The purpose of qualitative interviews is to diverge to all possible customer outcomes. The purpose of quantitative interviews is to converge on those outcomes customers desire the most.

Qualitative interviews help you avoid errors of omission: failing to uncover unarticulated customer needs. When you ask quantitative questions for interviews, you avoid errors of commission: choosing the wrong customer outcomes to improve with your new product.

There are many ways to collect these customer outcomes, but for B2B markets, the leading approach today is the Discovery interview. You record customer comments in sticky notes that they can see… using a web-conference or conference room projector. For more on this method, see the 5-minute video, Reinventing VOC for B2B.

As shown in the illustration above, you’ll ask the customer to identify two special types of outcomes:

• Top Picks (TP): These are just a few outcomes customers select at the end of the interview that they are most eager to see improved.
• Must Haves (MH): These are “price of entry” or “table-stakes” outcomes, without which your new product could not be used.

Step 3. Isolate key outcomes for quantitative interviews

It’s common to uncover dozens of customer outcomes in your qualitative Discovery interviews. You can’t ask B2B customers to help you prioritize all of them, because you’ll typically only have 1-to-2 hours of interview time before their patience runs out. For many B2B industries, it’s common to focus your quantitative questions for interviews on just 10 outcomes.

How does your team pick these 10? Follow these four rules:

• Only consider outcomes that were selected by customers as a Top Pick.
• Favor outcomes that were cited by many interviewees as Top Picks.
• Select outcomes that deliver obvious economic value to customers.
• Let each team member silently vote for their preferred outcomes first.

The last rule helps your new-product team avoid “group think” in selecting quantitative questions for interviews. Give each team member time to develop his or her unique perspective, and then share it with the group. You’ll have much better—and democratic—decision-making this way.

Step 4. Gather 1-to-10 Importance and Satisfaction ratings

Your two most important questions for your quantitative interviews are these:

• How important is (this outcome) to your company?
• How satisfied are you with your ability to (deliver this outcome) today?

You’ll ask these two questions for each outcome. As with Discovery interviews, it’s best to let customers watch as your record their input. You can do this using Microsoft Excel or Word. In the illustration below customer input is gathered during a “Preference interview” using Blueprinter^® software designed for this purpose.

You should “anchor” your numerical ratings with brief descriptions. In this example, the customer felt maximizing hiding power was “very important” and their current satisfaction was “barely acceptable… so ratings of “7” and “5”, respectively, were recorded.

It helps to let customers review their responses at the end of the interview, to see if a) they want to make any modifications or b) add outcomes you didn’t ask about. See below.

Step 5. Isolate a single market segment

Wait… didn’t we already do this in step 1? Maybe. You thought you had targeted a single market segment, but the only way to be sure is to analyze the answers to your quantitative questions for interviews. You do this by examining the standard deviations for the Importance and Satisfaction ratings you gathered.

Imagine you see the chart on the left (below) when you look at all interviews. Those large standard deviations should make you nervous, because this group of interviewees fails the test of a single market segment: “a cluster of customers with similar needs.”

At this point, you begin testing different sub-groups of interviews and generate a chart like the one on the right with tighter standard deviations. Feel better? You should, but only if you have strong rationale for why you grouped interviewees as you did for the right chart.

For B2B markets, there are typically three acceptable reasons for grouping interviewees into a “tighter” market segment…

• Different geography: Customers in Europe might have different needs than those in N. America or Asia, for instance.
• Different point in value chain: You might see differing needs along the value chain, e.g. paint producers vs. paint contractors.
• Different business model: Perhaps some paint producers sell directly to homeowners and others sell to paint contractors.

Step 6. Plan your new product design

Once you’ve isolated a single market segment, it’s time to analyze the fruit of your quantitative questions for interviews. A good first step is to plot each outcome’s average market Importance vs. Satisfaction. This is typically a simple average of all interview results in fragmented markets. For concentrated markets—with just a few buyers—you can weight interview results by each company’s market buying power.

The market will be most eager to see outcomes in the upper-left improved… those that are both important and currently unmet. In fact, your new product won’t be able to get a price premium unless it improves something important to customers that they were unsatisfied with.

Your quantitative questions for interviews can also deliver a Market Satisfaction Gap (MSG) chart as shown below. MSG is simply the average Importance times Dissatisfaction, where the latter is defined as 10 minus the Satisfaction score. So if paint Hiding Power had an Importance score of 9 and Satisfaction score of 6, MSG = 10 x (10-6) = 36%.

Our work with hundreds of new-product teams around the world has shown that Gaps in excess of 30% indicate high market eagerness for change. This is where your quantitative questions for interviews really pay off: When you begin the development stage you’ll know exactly what to work on and what to skip.

This is where your quantitative questions for interviews really pay off.

Step 7. Avoid forced ranking in interviews

Let’s talk about some poor quantitative questions for interviews. We’re often asked if forced ranking should be used. The logic goes like this: “Customer might rate everything a “10” in importance, so let’s ask them to rank outcomes #1, #2, #3, and so on.” If you anchor your numerical ratings as we showed earlier, this is just an imagined problem: B2B customers typically do a fine job of using the full ratings scale.

Forced ranking creates a major problem most users miss. With forced rankings, your new-product team will work on the highly ranked outcomes and avoid those ranked low, right? This can create two problems:

• Case 1: We’ve seen markets where every single outcome scored a MSG above 30%. All of these outcomes were good candidates to pursue… but forced ranking would lead you to ignore the outcomes with lower scores.
• Case 2: We’ve also seen markets where all outcomes had outcomes well below 30%. In this case the market is “over-served” and customers only want a lower price. But a team using forced ranking would enthusiastically develop a product around outcomes with the highest rankings compared to the rest.

Step 8. Add 3 more powerful questions

You know that “how important” and “how satisfied” are the most important quantitative questions for interviews. But we’ve developed three more that you can use for each outcome…

• How would you measure if this outcome is satisfied?
• Using this test, what result would be barely acceptable?
• Using this test, what result would make you totally satisfied?

Notice the use of the terms “barely acceptable” and “totally satisfied” above. If you refer back to the anchoring text for our 1-to-10 Satisfaction ratings, you’ll see that these represent a “5” and a “10” on the Satisfaction scale.

So in essence, you’ve asked customers to use your “satisfaction ruler” to measure how satisfied they are today. Then you calibrated your “ruler” by asking them what test results represent a “5” and a “10” on this ruler.

With this information, you are now able to replicate your customers’ experience on your own. You can test competitive products, your existing products, and—this is key—your new product design.

You are now able to replicate your customers’ experience on your own.

Is this important? You bet. Any price premium your new product garners will be based on the value it delivers over and above customers’ next best alternative. If you can’t quantitatively assess the difference between your new product and competing products, you’re limited to guessing at the optimal pricing.

Next steps…

In our experience, asking quantitative questions for interviews is the single most important thing B2B companies can do to boost innovation success. Doing so eliminates most commercial risk in the front end of innovation. This lets your scientists and engineers focus on reducing technical risk in the development stage.

Want to learn more about quantitative questions for interviews? Check out these resources:

1) View a dozen AIM case stories, showing how New Product Blueprinting was applied in a wide range of B2B markets. For each, you’ll see a Market Satisfaction Gap chart: These charts are fascinating because each tells a unique story about that market and its needs.

2) Check out our white paper on Market Satisfaction Gaps, with guidance on how to implement quantitative interviews and analyze their results.

3) Join us at one of our virtual public workshops. In two half-days, you’ll learn the world’s leading B2B voice-of-customer methodology… and role-play these customer interviews under the guidance of an AIM coach.