The Way We Do Customer Discovery is About to Fundamentally Change
If you routinely interview customers before building a new product or rolling out a major feature, your process might look something like this:
Kickoff a study
Plan to interview 10-15 people over two weeks
Run interviews / take lots of notes
Circle back with the team to make sense of your notes
Look for patterns and prioritize problems worth solving
Formulate possible solutions, test, build, measure, learn…
So what’s wrong with this picture?
The challenge with any kind of qualitative learning is prematurely jumping to the wrong conclusions. Why does this happen?
Here are some of the top reasons:
Unless two people run the interview, it’s hard to both lead the conversation and capture all the insights as notes.
Even if there are two of you, there isn’t a standard way to take notes, which makes them subjective and prone to the worldview and biases of the note-taker.
If there’s a disagreement on a particular insight, there is no way to audit the conversation.
And these are just the challenges with capturing insights after an interview. When we attempt to mine these insights for patterns, new challenges emerge:
We unconsciously look for evidence in the notes to validate the problem/solution combo (aka feature) already in our heads. Yes, this is our Innovator’s Bias at work.
Apart from the Innovator’s Bias, we easily fall prey to recency bias and other forms of confirmation bias, where we pay disproportioned weight to parts of the conversation that align with our already-formed worldviews.
The cost of prioritizing the wrong problem is wasting needless time, money, and effort building something only to realize a meh response post-launch.
Prioritizing the right problem(s) worth solving is too critical to leave to chance.
Earlier this year, we set our sights on improving the current customer discovery process at LEANSTACK.
Here’s what we came up with.
Part 1 of the Solution: Transcription
Part 1 of the solution was simple: Record and transcribe all your interviews. Transcribing an interview used to cost $1/minute 15 years ago. Today, you can transcribe a one-hour interview for $1. That’s a 60x improvement!
Most customer conversations already happen virtually, making it relatively easy to record the conversation (with permission, of course).
Some obvious advantages of transcription are:
It relieves the pressure of having to take great notes in real-time.
It captures the single source of truth (aka the voice of the customer), which makes your insights auditable.
As it can be heard/read by your entire team (accessible), it helps keep individual biases in check.
Some non-obvious advantages of transcription are:
It can be consumed much more quickly as we read faster than people speak.
But what I’m most excited about is that it can be mined much more deeply, resulting in more actionable insights.
One of my favorite features we tinkered with was utilizing machine-driven sentiment analysis to score and visualize an interview. Sentiment analysis works by scoring what the customer said into positive, neutral, and negative statements.
So something like “I like shopping at that store” scores positive, and something like “I hate pink” scores negative.
We use these scores to visualize the emotional shape of the conversation and to compute an overall sentiment score. For more on this, see The Simple Shapes of Customer Stories.
To be clear, all this happens automatically (machine-driven), and less than five minutes after uploading an interview, you get a picture that looks like this:
The first place I focus on is the overall sentiment score. Does it correlate with my assessment of customer satisfaction? If not, why not?
The visual shape of the conversation helps in pinpointing high and low moments in the conversation (emotion or energy) for deeper study. This is often where one finds spaces for innovation.
Part 2 of the Solution: Clustering
Part 2 of the solution came from realizing that machines are much better at insights mining than we are.
With transcripts in hand, could we surface patterns more reliably — free of cognitive biases using machine-driven learning (ML) and statistical analysis?
While ML is improving by leaps and bounds every day, we fell short on driving toward semantic meaning and context with the transcripts alone mostly because customer conversations like these tend to be intentionally unstructured.
We needed a way to distill a 45-minute conversation into a structured one-page summary.
Part 3 of the Solution: Mad-libs
Much like in chess, the answer came from recruiting a human-machine collaboration for the best results. The good news is that all customer journeys typically follow a universal story arc, which means there is a pattern.
We took inspiration from job stories, first popularized by the product team at Intercom and Alan Klement:
…and extended it to write out an entire 3-act Customer Forces Story Mad-lib.
We found mad-libs highly effective at teaching others the pattern while providing sufficient guard rails for staying within the boundaries of the pattern.
This opened the door to machine-driven insights mining.
The Proof is in the Pudding
Like we test all our products, we tested these solutions in our Problem Discovery Workshops, where we tasked attendees to uncover why people buy headphones. They interviewed a dozen people and transcribed and encoded their findings into a set of Customer Forces stories.
We then ran these 1-page customer forces stories through cluster analysis to reveal five distinct segments of customers, which we further reduced to three main segments.
Unlike demographical segments, job segments aim to group people based on behavioral similarities.
What about problem prioritization?
Problems, not solutions, create spaces for innovation. And the best way to search for problems is to search for struggle. We already have two inputs that correlate with struggle: customer satisfaction and sentiment score.
Plotting customer satisfaction and sentiment is a failsafe way for prioritizing the right job segments to mine. Sure, while the end goal is going a level deeper into specific problems, identifying the customer segment (or job segment) to tackle with the most struggle is the requisite first step in the process.
In a future post, I’ll cover how to further home in on problems worth solving from here.
Encouraged by our early experiments, we are in the process of rolling out a new customer discovery tool: Customer Forces.
If you currently conduct customer interviews (jtbd interviews or switch interviews) and would like early access, you can request early access here.