A Guide To Validating Product Ideas With Quick And Simple Experiments

12 min read
UX, Tools, Techniques, UX, User Research
Share on Twitter, LinkedIn

About The Author

Grace is the Co-Founder of Javelin, enterprise software for product teams to test the viability of new ideas. She has trained executives from GE, ESPN, and … More about Grace ↬

Email Newsletter

Weekly tips on front-end & UX.
Trusted by 200,000+ folks.

If you are building a product, you should always speak with customers and test your idea before. But you probably don’t know that *you *might be making some of the most common mistakes when running your experiments. Mistakes include testing the wrong aspect of your business, asking the wrong questions and neglecting to define a criterion for success. In this article, Grace Ng will show you a guide to designing quick, effective, low-cost experiments.

You probably know by now that you should speak with customers and test your idea before building a product. What you probably don’t know is that you might be making some of the most common mistakes when running your experiments.

Mistakes include testing the wrong aspect of your business, asking the wrong questions and neglecting to define a criterion for success. This article is your guide to designing quick, effective, low-cost experiments.

A Product With No Users After 180 Days

Four years ago, I had a great idea. What if there was a more visual way to learn about our world? Instead of typing search queries into a text field, what if we could share visual queries with our social network and get information about what we’re looking at? It could change the way people search! I went out, found my technical cofounder, and we started building a photo Q&A app. Being a designer, I naturally thought that the branding and user experience would make the product successful.

Six months later, after rigorous usability testing and refinement of the experience, we launched. Everyone flocked to the app and it blew up overnight. Just kidding. No one cared. It was that devastating moment after a great unveiling when all you hear are crickets chirping.

Confused and frustrated, I went back to the drawing board to determine why this was. My cofounder and I parted ways, and, left without technical expertise, I decided to step out and do some research by interviewing potential users of the app. After a few interviews, the root cause of the failed launched finally dawned on me. My beautifully designed solution did not solve a real human need. It took five days of interviews before I finally accepted this truth and slowly let go.

The good news for you is that you don’t need to go through the same pain and waste of time. I recently started working on another startup idea. This time, I followed a structured process to identify key risks and integrate customer feedback early on.

A Product With 16 Paying Customers After 24 Hours

I work with many entrepreneurs to help them build their companies, and they always ask me for feedback on the user experience of their Web and mobile apps. They express frustration with finding UX talent and want some quick advice on their products in the meantime. This happens so frequently that I decided to learn more about their difficulties and see what might solve their problems.

I specified what I was trying to learn by forming a hypothesis:

“Bootstrapped startup founders have trouble getting UX feedback because they have no reliable sources to turn to.”

To test this, I set a minimum criterion for what I would accept as validation to further explore this opportunity. I had enough confidence that this was a big problem to set a criterion as high as 6 out of 10. This means that 6 out of the 10 people I interviewed needed to indicate that this was a big enough problem in order for me to proceed.

By stating my beliefs up front, I held myself accountable to the results and reduced the influence of any retroactive biases. I knew that if these entrepreneurs already had reliable sources to turn to for UX feedback and that they were happy with them, then generating demand for an alternative solution would be much harder.

Design of my first experiment on the Experiment Board. (Large preview)

In three hours of interviews, I was able to validate the pain point and the need for a better alternative. (You can watch a video walkthrough of my findings.) My next step was to test the riskiest assumption related to the solution that would solve this problem. Would they pay for an online service to get feedback from UX designers?

Instead of building a functioning marketplace with designer portfolios and payment functionality, or even wireframing anything, I simply set up a landing page with a price tag in the call to action to test whether visitors were willing to pay. This is called a pitch experiment. You can use QuickMVP to set up this kind of test.

Test if customers will pay for the service. (Large preview)

Behind the landing page, I attached a form to gather information on what they needed help with. Within a few hours, 10 people had paid for the service, asking for UX feedback on their websites. Having validated the demand, I needed to fulfill my promise by delivering the service to the people who had paid.

Did not build any functionality; just a form to collect information. (Large preview)

Because this market is two-sided — with entrepreneurs and designers — I tested the demand side first to see whether the solution provided enough value to elicit payment. Then, I tested the supply side to learn what kind of work designers were looking for and whether they were willing to consult virtually.

Test the second side of the market: the supply side. (Large preview)

To my surprise, the UX designers I spoke with had established clientele and were very picky about new clients, let alone wanting to consult with random startups online. But they all mentioned that they had been eager to take on any work when they were first starting out and looking for clients. So, I switched my focus to UX designers who are not yet established and are open to honing their skills by giving feedback online.

Armed with these insights, I iterated on my landing page to accommodate both sides of the market and proceeded to fulfill the demands of the customers I had accumulated. No wireframes. No code. Just a landing page and two forms!

A landing page that tests both sides of the market, simultaneously. (Large preview)

Interest in this service can also be measured by their willingness to fill out the form. (Large preview)

To simulate the back-end functionality, I emailed the requests to the UX designers, who would then respond with their feedback, which I would email back to the startup founders. Each transaction would take five minutes, and I did this over and over again with each customer until I could no longer handle the demand.

Do things that don’t scale in order to acquire your earliest customers and to identify business risks that you might overlook when implementing the technical aspects. This is called a concierge experiment. Once the manual labor has scaled to its limit, then write the code to open the bottleneck. Through this process, I was able to collect feedback on use cases, user expectations and ideas for improvements. This focused approach allowed for more informed iterations in a shorter span of time, without getting lost in wireframing much of the application up front.

Today, BetterUX.it is a service through which startup founders connect with UX designers for feedback on their websites and apps, and designers get paid for their expertise and feedback.

How To Create A Product That People Want

What did I do differently? The structured process of testing my assumptions saved me time and confirmed that each part of my work was actually creating value for end users. Below is a breakdown of the steps I took, so that you can do the same.

Should You Build Your Idea?

My first mistake with my first startup was assuming that others had the same problem that I experienced. This is a common assumption that many gloss over. Build products that scratch your own itch, right? But many entrepreneurs realize too late that the problems they’re trying to solve are not painful enough to sustain a business.

As product people, we often have many ideas bubbling in our heads. Before getting too excited, test them and decide which one is the most viable to pursue.

Design An Effective Experiment

To get started, break down your idea into testable elements. An effective experiment must clearly define these four elements:

hypothesis,
riskiest assumption,
method,
minimum criterion for success.

At Lean Startup Machine, we’ve created a tool called an Experiment Board, which enables us to easily turn crazy ideas into effective experiments in a few minutes. As you go along, use the board as a framework to design your experiment and track progress. Refer to the templates provided on the board to quickly formulate your hypothesis, riskiest assumption, method and success criterion.

Construct A Hypothesis

Every experiment starts with a hypothesis. Start by forming a customer-problem hypothesis. Once it is validated, you can go on to form a problem-solution hypothesis.

Define your customer. Which customer segment experiences the most pain? They are your early adopters, and you should target them first. These are the people who have the problem you’re solving for, and they know they have the problem and are trying to solve it themselves and are dying for a better way! Most people have trouble identifying these customers. If you do, too, then just segment your potential customer base by level of pain and differentiating characteristics, such as lifestyle and environmental factors. Being specific will reduce the time it takes to run through experiment cycles; once you’ve tested against that one segment and found that the problem doesn’t resonate with them, you can quickly pivot to test another customer segment. In the long run, having a clear idea of who you’re building for will help you maintain a laser focus on what to prioritize and what to dismiss as noise.
Define the problem. What problem do you believe you are solving for? Phrase it from your customer’s perspective. Too often, people phrase this from the perspective of their own lofty vision (“The Web needs to be more human”) or from a business point of view (“Customers don’t use our service enough”). Also, avoid being too broad (“People don’t recycle”). These mistakes will make your hypothesis hard to test with a specific customer, and you’ll find yourself testing for a sentiment or an opinion in interviews, rather than a solvable problem. If you have trouble, phrase the problem as if your friend was describing it you.
Form a hypothesis. Brainstorm on a few customers and problems to consider all the possibilities. Then, combine the customer and problem that you want to focus on with this sentence: “I believe customer x has a problem achieving goal y.” You have just formed a testable hypothesis!

Identify Your Riskiest Assumption

Now that you have formed a customer-problem hypothesis, poke some holes and extract the riskiest assumption to be tested. Start by brainstorming on a few core assumptions. These are the assumptions that are central to the viability of your hypothesis or business. Think of an assumption as the behavior, mentality or action that needs to be true in order to validate the hypothesis.

Ask your team members, boss or friends to suggest any assumptions that you may have overlooked. After listing a few, identify the riskiest one. This is the core assumption that you are most uncertain about and have the least amount of data on. Testing the riskiest assumption first will speed up the experiment cycle. If the riskiest assumption is invalidated, then the hypothesis will be invalid, and you will have saved your company from going down the wrong path.

Choose A Method

After identifying the most critical aspect of your idea to test, determine how to test it. You could conduct three kinds of experiments before getting into wireframes. It’s best to start by gathering information firsthand through exploration. But you could choose a different method depending on your level of certainty and the data.

Exploration Conduct qualitative interviews to verify and deepen your understanding of the problem. Even though you experience the problem yourself, you don’t know how big it is or who else has it. By conducting exploratory interviews first, you might realize that the opportunity isn’t as big as you had thought or that a bigger problem could be solved instead.
Pitch Make sure the solution would actually provide value by selling the concept to customers before building the product. This will measure their level of determination to solve the problem for themselves. A potential customer not taking a certain action to use your service, like paying a small deposit or submitting an email address, indicates that the problem is not painful enough or that you haven’t found the right solution.
Concierge Personally deliver the service to customers to test how satisfied they are with your solution. Did your value proposition meet their expectations? What was useful for them? What could have been done better? How likely are they to return or recommend the service to a friend? These are all insights you can discover in this step.

Set A Minimum Criterion For Success

Before running the experiment, decide up front what result will constitute success and what result will constitute failure. The minimum criterion for success is the weakest outcome you will accept to continue allocating resources and pursuing the solution. Factors like budget, opportunity cost, size of market, level of demand and business metrics all play into it.

The criterion is usually expressed as a fraction:

“I expect x number of people out of the y number of people in the experiment to exhibit behavior z.”

I like to set the criterion according to how big I think the problem is for that customer segment, and then determine how much revenue it would have to generate in order for me to keep working on it. At this point, statistical significance is not important; if your target customer segment is very specific, then testing with 10 of them is enough to start seeing a pattern.

Once you have validated the hypothesis with a small sample of the customer segment, then you can scale up the experiments to test with larger sample sizes or with other segments.

Run The Experiment

Once you have defined these elements, you are ready to run the experiment! Have team members look at your Experiment Board and confirm whether they agree with what you’re testing. This will hold you and the team accountable to the results, so that there are no subjective arguments afterwards.

Analyze The Results And Decide On Next Steps

After gathering data from your target customers, document the results and your learning on the Experiment Board. Did the results meet your criterion for success? If so, then your hypothesis was valid, and you can move forward to test the next risk with the product. If not, then you need to form a new hypothesis based on your learning to get closer to something that holds true. Track your progress over time on the Experiment Board to get a holistic picture of your validated learning and to continually make informed decisions.

Test and repeat. You’re on your way to creating a great product that people want!