Experimenting in Online Stores with direct feedback from our customers – using Learning Launches

Challenge & Selection

At our online store, we have electronic software/services distribution (ESD) available for our customers  in Latin America. One of the challenges is understand which approach on products presentation, page  layouts and content, tend to convert better with our customers. The tool that I used was Learning  launch to test ideas proposed. In the company, we need to experiment, learn and put as soon as  possible the best idea to work due the dynamic of the business. The business objective is to increase  sales of gaming memberships (like Xbox Live, EA Access or PSN memberships) through our Online Store,   our main audience are gamers and the traffic to the site allows us to get statistical confidence in few  weeks which is really good to learn and improve quickly from the experiments.

Application

First of all, we evaluated the current data that we had from our gaming section in the Online Store. The  customer journey seemed healthy almost over the year: the traffic was high, the bounce rate low than  the average and the conversion was three times over the average of the site. However we started to  receive some feedback (post-purchase verbatims) that suggested that our layout could improve: even  with high sales, sometimes customers opined that they found the journey very large to find the  subscription: many clicks and many modules in the page. Also some people wanted to understand how  to get the most from their gaming memberships. In the site everything was around to sell since we  thought that the memberships were already well known and the sales “proved” our theory. But  improvements maybe prove that we can increase and engage more users.

We generated several layouts, most of them with well-proven storytelling modules that worked in other  categories. We showed first to a small group that we have contact through our product managers, these  people could give us more insights about them and help us to filter for the best ones. In that phase,  gamers group and the team worked to get the two best ones to test against the current layout. One  discovery in that phase was that the selected layouts from groups were more focused in lifestyle or  games images, more detailed benefits text and CTAs (call-to-action) than the traditional modules:  clickable product image with generic description.

Then we tested in the platform the control versus the two challengers. The control was more like  generic product presentation layout, the challenger one a full benefits storytelling modules and the  challenger two a mix of both layouts. For the test also we put in place a post purchase survey to get  more opinions about the gamer experience. With that, we not only get the best idea. The feedback was  good and also gave us more ideas around the gamer experience in our Online Store. Both phases, the  “internal” with the team and gamers group were helpful to decide which layout could meet customer  need at the purchase moment and generate more conversions.

Insights & Approach

It was the first time we conducted a test in that way, with many phases and very valuable input from our  customers and third party point of view (gamers group). Usually we design the layouts, put in the  platform and focus in kpis to select the best one, but just based on what we think and expect, especially

for revenue, not on customer thoughts and needs. This last part I emphasized our team that in  mid/long-term will give us great results. The layout chosen was more about the mix of storytelling + the  traditional e-commerce modules and now we see more engagement in the page and also the initial  conversion increased by 0.5pts. which is a good signal of the experiment. Also with the feedback we  found new opportunities to improve in other stages of the customer journey.

The main thing for these type of experiments is to find the right time to deploy them. As mentioned the  dynamics of e-commerce sometimes don’t allow to take much time. But the team is now convinced that  finding right periods and get direct customer feedback than experiment kpis is a better approach.


Experimenting in Online Stores with direct feedback from our customers – using Learning Launches Challenge & Selection At our online store , we have electronic software / services distribution ( ESD ) available for our customers in Latin America . One of the challenges is understand which approach on products presentation , page layouts and content , tend to convert better with our customers . The tool that I used was Learning launch to test ideas proposed . In the company , we need to experiment , learn and put as soon as possible the best idea to work due the dynamic of the business . The business objective is to increase sales of gaming memberships ( like Xbox Live , EA Access or PSN memberships ) through our Online Store , our main audience are gamers and the traffic to the site allows us to get statistical confidence in few weeks which is really good to learn and improve quickly from the experiments . Application First of all , we evaluated the current data that we had from our gaming section in the Online Store . The customer journey seemed healthy almost over the year : the traffic was high , the bounce rate low than the average and the conversion was three times over the average of the site . However we started to receive some feedback ( post - purchase verbatims ) that suggested that our layout could improve : even with high sales , sometimes customers opined that they found the journey very large to find the subscription : many clicks and many modules in the page . Also some people wanted to understand how to get the most from their gaming memberships . In the site everything was around to sell since we thought that the memberships were already well known and the sales “ proved ” our theory . But improvements maybe prove that we can increase and engage more users . We generated several layouts , most of them with well - proven storytelling modules that worked in other categories . We showed first to a small group that we have contact through our product managers , these people could give us more insights about them and help us to filter for the best ones . In that phase , gamers group and the team worked to get the two best ones to test against the current layout . One discovery in that phase was that the selected layouts from groups were more focused in lifestyle or games images , more detailed benefits text and CTAs ( call - to - action ) than the traditional modules : clickable product image with generic description . Then we tested in the platform the control versus the two challengers . The control was more like generic product presentation layout , the challenger one a full benefits storytelling modules and the challenger two a mix of both layouts . For the test also we put in place a post purchase survey to get more opinions about the gamer experience . With that , we not only get the best idea . The feedback was good and also gave us more ideas around the gamer experience in our Online Store . Both phases , the “ internal ” with the team and gamers group were helpful to decide which layout could meet customer need at the purchase moment and generate more conversions . Insights & Approach It was the first time we conducted a test in that way , with many phases and very valuable input from our customers and third party point of view ( gamers group ) . Usually we design the layouts , put in the platform and focus in kpis to select the best one , but just based on what we think and expect , especially for revenue , not on customer thoughts and needs . This last part I emphasized our team that in mid / long - term will give us great results . The layout chosen was more about the mix of storytelling + the traditional e - commerce modules and now we see more engagement in the page and also the initial conversion increased by 0.5pts . which is a good signal of the experiment . Also with the feedback we found new opportunities to improve in other stages of the customer journey . The main thing for these type of experiments is to find the right time to deploy them . As mentioned the dynamics of e - commerce sometimes do n’t allow to take much time . But the team is now convinced that finding right periods and get direct customer feedback than experiment kpis is a better approach .