Netflix on Making a Bigger Bet on Incrementality | AppsFlyer


Why Netflix’s Data Scientist Jeffrey Wong Wants the Digital Ad Industry to Make a Bigger Bet on Incrementality

October 7, 2020



Jeffrey Wong, Principal Data Scientist, Computational Causal Inference at Netflix discusses how the streaming giant uses data signals to help get you to notice – and hopefully watch – its shows. Wong also breaks down the company embracing the measurement of ad spend effectiveness through incrementality and why he’s urging more marketers to take risks with their advertising budgets.



Mike Shields (19s):
Hey guys, this is Mike Shields and this week on Next in Marketing we got a really interesting, unique guest, Jeffrey Wong, Principal Data Scientist of Computational Casual Inference at Netflix. Yes. That’s his real title. We talk to Jeff about how Netflix uses all of its unique data and its willingness to experiment to get you sucked into shows that you are almost guaranteed to love. Plus, we went deep into Netflix’s experimentation platform and why big brands need to embrace measuring incrementally when they figure out where to spend their budgets. Let’s get started. Jeff, how are you? Thanks for being here.

Jeffrey Wong (51s):
I’m doing great. Thank you so much for hosting.

Mike Shields (51s):
No, thanks to you. And I’m excited to talk to somebody at Netflix and this is not a typical subject matter for me or this podcast. You are one of the more unique, unusual titles I have seen, but let’s, let’s start with— you work in Netflix’s experimentation platform, which sounds really cool. Tell us what that is and what it’s like.

Jeffrey Wong (1m 11s):
So being experimentation driven is a huge part of Netflix culture and we use experimentation and in many aspects of that business in order to improve operations and kind of ultimately member joy and the experience that users have, and that can range from improvements to the streaming experience or the UI. And then of course it’s probably most relevant today to today’s topic in marketing.

Mike Shields (1m 39s):
So, okay. You’re not talking about programing when you talk about the experiments or an experimentation platform, obviously, although you have a culture that is about trying new things and taking risks. But are we talking about like just tweaking the way that the shows are presented to individuals and the way that the UI lays out, what are different viewing options or the recommendation algorithms? What things do you guys experiment with?

Jeffrey Wong (2m 3s):
I think we do experiments in pretty much all parts of the business and so a couple of examples are things like what type of artwork that we show. We have a public Netflix tech blog that kind of discusses how we think about how to pair content, TV shows, movies with different types of artwork that will be able to attract people to shows that that they might be interested in. We also do experiments to figure out how to best optimize the layout of the page for different devices and how to optimize the slate of content that we recommend for users.

Mike Shields (2m 49s):
So that reminds me of when you’d read about how Google—in the early days would spend like an enormous amount of time trying to figure out what color the links should be, what font they should use, because there was a real science to what people are responding to. So it’s not—I think people wonder with Netflix if you just show up, throw up a library with the marketing photos, from all these different shows that works. But no, you guys—it sounds like you guys tweak and play around with, you know, what images are people going to respond to and different different users will respond to different faces and actors and stills and stuff like that?

Jeffrey Wong (3m 20s):
Yeah. I think there’s a really fun example with with Stranger Things that I like talking about. In Stranger Things, it’s a huge hit show that we have on the service, and there are a couple of different ways that you could potentially market it to someone. You could say that it’s a thriller, or you could say that it’s a show about kids. You could also say that in a very specific way. You could probably say that its kind of this show with a throwback feeling to what life was like in the eighties and, in particular, there’s this one image that is really interesting to talk about.

Jeffrey Wong (4m 2s):
It’s this image where the kids are dressed up in their Ghostbusters costumes and they are going to school. It’s Halloween. They are going to school in their Ghostbusters costumes. And we kind of want to figure out whether or not that’s a good image that people will relate to. Obviously, if you’ve never seen Ghostbusters, you might not even be able to relate to this part of the show. But, if you have seen Ghostbusters, maybe you will relate to that part of the show.

Jeffrey Wong (4m 32s):
Ghostbusters was a movie that was produced in the U.S., but Stranger Things is streamed all around the world. And so, we also kind of have to figure out in an international audience, do people relate to Ghostbusters, which again was an American production. And so we have to ask all of these types of questions and try out different types of images to figure out what kind of experience is—what kind of images will give the best and richest experience for the members.

.  .  .

Click to download the full transcript

Meet our host, Mike Shields

Mike Shields is the founder of Shields Strategic Consulting. Shields covered the ad business for over 15 years at top publications including The Wall Street Journal, Business Insider, Adweek and Digiday.

Do you have a compelling idea for a story? Interested in joining the conversation? Have a suggestion for the next guest?