User Study Realness (The Grass Roots Edition)

I’ve done user studies for several years but I’m excited about getting the opportunity to implement user studies following the lean startup philosophy.

From Laura Klein’s book UX for Lean Startups, “Lean UX is solidly data driven.” In other words, you test first, assume never.

I should set the stage now and share with you that I love looking back on user studies I’ve done in the past. Especially those that were purely grassroots. Really grassroots. Let me introduce you to to one of these gems.

Right off the bat, I can tell you how we wrongly went about this study: we didn’t first form a hypothesis that we were trying to answer.

Here’s what we assumed: our website messaging wasn’t clear, the website flow made no sense and our checkout system was confusing.

Basically we went into the user study looking for validation for what we wanted to do, which was completely rework the website and messaging.

I joined the startup after the website had launched but I wished this study had been done prior to launching the site. Again, that’s where the lean startup school of thought makes so much sense.

We started by interviewing 16 people, all non-users of our product. With the exception of 4, they had never heard of us.

We had them walk through our booking process, from the homepage to choosing a product they were interested in, booking and reserving that product. Finally, we asked them about a couple of specific pages we felt were lacking in clarity and flow.

“$60 is too much. What am I getting for that?”


The user study validated a lot of what we already knew, like:

Messaging: It wasn’t clear to the user what exactly we were offering or where they could have the meal. Would the chef come to their house? Was it at the chef’s house? Was it delivery? The user really didn’t know. Dynamic Pricing: The need for pricing that changed depending on how many people you have at a dinner. The price displayed isn’t what you would pay if there was only 2 people…or a larger group like 15. Users were unaware that booking a dinner under the minimum number of people shown would result in a higher price per person. Flow: From the homepage to the meals page, filtering down to what you wanted to see (cooking classes, meals, tours) wasn’t intuitive. Part of that stemmed from hazy messaging.

“I thought you served only Italian food.”

New Insights:

We also got a lot of great feedback on issues we didn’t know were issues:

Consistent Images: One thing we heard was about our H1 image on our homepage. It conveyed (to a couple of users) that we offered one genre of food. Of course, there’s no way a picture of food will convey every genre but still, this was interesting feedback. If only to reiterate how important visuals are to our site. Menu Clarity: We have pictures of food and a menu listed on the meals page but you’ll notice they’re not listed side-by-side. We heard from some users that this added to their confusion, especially when the menu featured food they were unfamiliar with. Social Trust: We offer Facebook login but not using the Facebook symbol (it simply says “Login”). The study showed us that users wanted to see the Facebook symbol. To them, it added validity and trust. Next Steps:

The raw feedback was enlightening enough but when it came to actionable items to provide backend, we focused on three areas:

  1. Messaging on the home and meals pages

  2. Dynamic pricing

  3. How it Works

(Our reasoning for putting How it Works as a high priority was because of heavy site traffic to that page.)

I would love to share the before and after images but my path took a quick turn while in the middle of this study. Instead, I handed it over to my very capable coworkers and look forward to seeing the end result as a user.