Challenge    I lead the product management team for an online marketplace that sells used cars. We connect  potential car buyers with car dealerships. Our site features thousands of cars. We allow buyers  to search for cars, apply relevant filters, browse car information (photos, car history etc) and  enquire on specific cars. We define success by the number of enquiries submitted by our  buyers.     Our daily challenge is to find new ways of getting car buyers to submit enquiries on cars. More  specifically, we came up with an idea which we needed to test. It was based on the following  hypothesis: Car buyers would be willing to provide their contact details (and enquire on a car) in  exchange for a free car history report​.     Selection    The most appropriate tool to test our hypothesis appeared to be a ​“learning launch”​. We knew  that building the full solution would turn out to be costly (reports are provided by a third party  and our team would have had to build technical integrations).     We decided to find the quickest and cheapest way to test our hypothesis. And if it turned out to  be true, we would then dedicate engineering and product resources to building the proper  solution.    Application     In order to test our hypothesis quickly and cheaply, we ran an A/B test on our website. We  created a very “light” version of the new feature (report download) and only showed it to a  portion of our users (group A). Other users (group B) remained on the current experience and  were not exposed to the report download feature.     We ran the test for 3 weeks and compared results between group A and B. We agreed that our  success metric should be “enquiry conversion” (% of users who end up submitting an enquiry on  a car). Group A enquiry conversion turned out to be significantly higher than that of group B  (+20%)!    This proved that our hypothesis was correct. From that point on, we knew that providing free  reports to car buyers would lead to more car enquiries. It also provided us with data to build a  business case and justify investing engineering and product resources to build the end solution.     It’s worth noting that our approach involved building a very light version of the feature. We opted  for a manual process: when a user would request a report, an email would be sent to our  customer service team who would purchase the report and email it back to them. Now that the

test has been successful, we can invest engineering resources to build a proper solution, having  our system automatically querying the supplier’s systems.    Approach    I believe the tool we selected (learning launch) was the right one. We started with a hypothesis,  ran a quick test and got data to validate our hypothesis.     That being said, there are a few things we could have done differently. During the test, we  compared two versions of the product: A (with the report download feature) and B (without the  report download feature). Besides the new feature, A also included several usability tweaks.  When the first set of results came through, we were unable to determine if the increase in  enquiries was solely due to the report feature.     We had to go back to the drawing board and re-run the test with the two versions (A and B)  identical from a usability perspective. This allowed us to have absolute confidence that the  increase in enquiries was due to people wanting to download reports.

