CX Spotlight: How Netflix Does Product Testing

The user experience (UX) design, prototyping, and collaboration platform, Invision recently hosted a series of “Design Talks” webinars. In each of these webinars, Invision invites UX teams from different companies to share their perspective, workflow, lifehacks, and knowledge in the UX design space.

My favorite guest presenter in the series was Zach Schendel, Director of UX Research at Netflix. Zach told the story of transforming data and insights gained from actual end users into UX design solutions that make Netflix a more engaging, more enriching experience.

In this blog post we’ll break down a few of our key takeaways and highlight ways the Netflix and Atypical approaches align. Let’s jump in.

Start at the End User

First, let’s start where most UX research starts – with the end user. When most people first think of UX design they often have visions of wireframes and user flows. That’s UX design, right? Well, it’s not wrong, but it’s not the whole story either. It’s the proverbial tip of the iceberg, the part of the work that’s often most visible. There’s much more below the surface.

Most of user experience,and the larger Consumer Experience discipline, is about research – getting into the minds of the end user, understanding their points of view, and empathizing with their frustrations. Without that first-hand knowledge our user flows are inherently flawed and wireframes will not yield excellent experiences we hope to be designing for.

When starting end-user research, you must define first your goals. For the Netflix UX team their goal is to maximize engagement and satisfaction. After all, in the subscription world, where retention is life, an easy to use, flow-state user experience is of critical importance.

Now, in order to increase engagement, we have to know how the thing is currently being used. Don’t settle for what the slick whitepaper said or what a competitor’s site is doing, or some fodder in a clickbaity article. You need real feedback from real users about your product; only then can you hope to make changes that are meaningful. Consider best practices, competitive parity, or subject matter opinion as inspirational but not instructional.

Zach and his team employ a host of primary research methods, from good ol’ engagement metrics and your standard-fare usage analytics, to advanced eye tracking and ethnographic studies where researchers watch people watching Netflix. And no I don’t mean that creepy/paranoid “they’re watching us,”, I mean it’s in a usability lab. Doing so allows them to understand exactly how users interact with the Netflix.

Takeaway: start with your end users and understand them, empathize with them, and become their advocates if you’re going to design a better, more engaging digital experience. After all, it’s their experience you’re designing for.

Change What Matters

Armed with not just data but insight into what actual Netflix end users are doing (and not doing), Zach and the rest of the Netflix UX researchers can start designing better user experiences.

One particular insight that Zach’s team uncovered was that the vast majority of Netflix TV users (the largest user group and focus of the webinar) rarely – almost never- scroll up.

So, we know users aren’t scrolling up, but why? “There’s just no affordance to scroll up,” said Zach, meaning this behavior has little to no value for users. This sort of makes sense; when someone is in discovery mode, they want to go in one direction and backtrack very little.

But what’s the problem? The menu, stuck at the top of the experience, is therefore almost entirely hidden, hold for first entry into experience. When engagement is of critical importance it’s no doubt the UX team at Netflix needed to find a fix.

Zach and team know how users interact with their product, and knowing why allowed Netflix to rethink navigation,turning it on it’s side, figuratively and literally. Rather than relying on a top navigation (a cue from website design) they went with main navigation tucked in a side pane, providing easy access to the menu, search functionality and a host of other navigation elements within the interface – no vertical scrolling required.

2013 Netflix TV UI Design

2018 Netflix TV UI Design

Takeaway: you can’t change what people are doing – you can’t make scrolling up a thing. But what you can do is design solutions to existing behaviors, wants, and problems in the digital experience.

Test and Test Again

The roll out of the above side pane, and other features included in the 2018 TV user interface (like full bleed video) were likely tested, vetted, and confirmed to have a positive (or at minimum net neutral) impact to user experience through A/B testing and experimentation.

But more than features at Netflix are being tested. Another insight generated from primary research of actual end users is that, when browsing for something new to watch (a common end user behavior), 91% of titles were being rejected (aka not watched) based on approximately 1 second of consideration. As mentioned, we can’t change what people are doing, we can’t make users look at box art longer, but what Netflix can control what they are looking at: the box art.

One example of this is “My Little Pony, Friendship is Magic” (great show, gotta watch). Users were given six different variations of box art (a control and five test variations), and what resulted was a 62% increase over control with one of the test variations. A clear winner. But it didn’t stop here.

What started as a fairly straightforward A/B/n testing practice at Netflix has now matured into a full personalization engine, serving targeted content to users with the goal of creating a better, more relevant experience.

There are now countless examples of this (well, I’m sure someone at Netflix is counting) but in the following example, rather than serving one box art of “Stranger Things” to all customers, Netflix served several variations to different audiences, with each variation catering to those distinct audience wants, interests, and preferences. Doing so is all with the goal of making for a more curious browser, a more engaged viewer, and more satisfied Netflix subscriber.

How does Netflix know what box art to serve you? Simple: you telling them. One of the largest indicators for relevance in box art is your previous viewing behavior. If you tend to watch programs within a certain genre or with a certain cast member, Netflix will do its best to serve you the most relevant image that caters to that behavior.

For some, this may sound a bit creepy. I’ve heard that reaction sometimes when discussing website experimentation and conversion rate optimization (CRO) with our clients.. I reassure them that our goal isn’t trying to trick people. It’s not hoodwinking, it’s not wizardry; it’s all about relevance. It’s about delivering value.

It’s trying to create not just the best experience, but the best experiences. As Zach put it (and I may have paraphrased, but the spirit is there): “‘Are you tricking people into watching show?’ No we are looking for improving quality to increase engagement.”

Takeaway: Digital experimentation is a win-win. It should serve the end user with a more delightful experience and serve the client with a more optimized engagement, better lead generation, or optimized conversion paths. We are moving away from the world of one-size-fits-all into custom-fitted digital experiences, and I for one embrace it.

Start Testing Like Netflix

To learn more about Invision or to watch the Netflix webinar for yourself, visit

To speak with one of our Atypical CX team members to learn how your company, like Netflix, can conduct end-user research, change what’s important, and experiment for both the benefit of the end user and the company – drop us a line.

No Comments

Post A Comment