1 Leonardo Torres

Reflection – Learning Launch

Challenge

In this reflection about learning launch and experimentation, I would like to tell an  incident when I influenced engineering to build a particular feature. In my current role,  I’m the monetization front-end designer for our free email product.

Two weeks ago, Irina, the engineering manager, gave me some bad news, "Leonardo,  I’m sorry, but we have to remove 140x500 skyscraper ad. My engineering VP told me  that the PM team wouldn’t mind."

I was thinking to myself: “Are you kidding me? Wouldn’t mind? This ad was only the 3rd  most valuable ad placement on the entire site. It delivered INR 45 million in revenue  per year. Who did Irina think was paying her salary?”

Selection

After I collected my emotions, I asked Irina, “When are you planning to remove it?”  She said, “Two code sprints from now. About 6 weeks.”

I told her, “Can you give me three days to pull some data? I’d like to understand the  impact of this decision.”

After she left, I thought: what could I do? I could have a hissy fit and complain loudly.  Or I could immediately escalate to the engineering VP.

While I was tempted to go with either or both solutions, I wouldn’t get what I want  unless I was armed with the facts. Before I figured my plan, I gave my boss Luca, a  heads-up. He wasn’t pleased with engineering’s unilateral response, but he agreed with  my plan.

Application

I logged into the data warehouse and pulled revenue numbers for the last 12 months.  Next I asked to sales team to gauge customer reaction if we no longer made the product  available. Lastly, I reached out to the VP Engineering’s Chief of Staff to better  understand why we wanted to remove the ad.

Insight

After I collected the data, I set up a meeting with Irina, Luca my boss, and  Luca the VP of Engineering. The VP of Engineering reiterated his decision to  remove the skyscraper ad. He maintained what I had heard earlier: the skyscraper ads  were the reason why the email product’s engagement and market share was down.

I asked him, “Why do you think the ads impact engagement?”

2 Leonardo Torres

He said, “They distract the user.”

I replied, “Are there times when you don’t find the ads distracting?”

He said, “Yes, when it’s relevant to the content. Let’s say I’m reading an email about  meeting with friends, and the ads suggest things new restaurants and bars.

So I summarized, “So, you don’t have a problem with ads. What you’re really concerned  about is the quality of the ads.”

Approach

I recommended that rather than remove the ads completely, we should revisit our ad  policy and make sure that we’re serving relevant ads. We can tighten up our ad editorial  processes so that low quality ads aren’t mindlessly approved by the system.

He was shocked when I told him that the ad placement generated $45 million in annual  revenue. And he also didn’t realize that by pulling the placement, customers would pull  advertising budget, primarily because it wasn’t worth continuing a relationship with us,  given the low ad volume and lack of attractive ad placements, relative to our  competitors.

Organisation

We minimized many of the low quality teeth whitening, weight loss ads that were  known for distracting tactics such as pop-ups, fake navigation, and blinking sections.

It required a lot more work on our editorial team and advertisers. Shortly after, we later  found that email user engagement did go up by 6.5%, and most importantly, we  preserved $45 million in revenue.

