Learning Launch Reflection

Challenge

One of the responsibilities of my role as online chat channel leader was to increase customer  satisfaction (C-SAT) scores.  Customer satisfaction is measured through surveys given to customers  immediately after the chat is over.  Agents can see the results of these surveys, but since the  information is available only after the chat is over, there is obviously no way for agents to go back and  improve upon that poor interaction.  They can only try to do better with the next customer.  Moreover,  supervisors can only see their agents’ scores as interactions are completed throughout the day.   Therefore, they can only coach to deficiencies after-the-fact.

While searching for tools and practices that might help increase customer satisfaction, I learned that  data scientists at my company had developed a Natural Language Processing (NLP) model that used  account notes to predict if a customer would file a regulatory complaint. I asked, “What if we used this  same to approach to analyze historical chat transcripts and uncover the correlation between certain  words/phrases and C-SAT scores?”  The data scientists quickly proved that such a model could be built  and was indeed highly predictive.  I then asked, “What if we predicted the C-SAT score in real-time so  that agents can better understand the customer’s emotional state and adjust their responses and so  that supervisors can intervene on the chats that are most at risk of ending with dissatisfaction?”

Selection

These “what-if” questions were intriguing – particularly because 1) our chat operation was staffed with  offshore, non-native English-speakers, 2) the agents handled up to three chats simultaneously, and 3)  the emotive content of written channels like chat or correspondence is naturally lower than in-person or  phone channels.  We hypothesized that our agents (or at least some subgroup) could improve their C- SAT scores if they had a visual, quantitative display of customer sentiment throughout the conversation.   At the same time, we worried the tool, if also given to supervisors, would put too much pressure on the  agents and cause them to take too long to complete normal requests.  Since such a tool had never been  used at the company, we had no prior data to help us assess the impact on C-SAT scores or on handle  time.  Therefore, it was fairly clear that we would need to run a pilot / learning launch.  The alternative  (which had been done far too often in the past) would have been to assume the idea was a good one  and to immediate start full-scale development.

A key assumption was that C-SAT scores are materially impacted by the degree of empathy shown by  the agent, as opposed to say, the policies of the company or the capability of the agents.  We knew from  experience all of these play a role, but it is nearly impossible to quantify the relative degree.  Another  assumption was that one or more of the factors above impeded the agents from empathizing with  customers.  There was a hard push from our analysts to set up a statistically significant A-B test to  measure impact to C-SAT scores and handle time, but this would have required significantly more work,  a longer time to market, and greater cost.  As James Angelo pointed out in the “Learning Launch”  video, it was critical to get learnings quickly and cheaply.  Therefore, we decided that the goal of the first

learning launch would be to gather qualitative feedback, and if it was favorable, we would then make  the additional investment in a true A-B test.

Application

We assembled a team that had visual design, software engineering, data science, and operations  expertise.  Our first step was to develop paper prototypes showing potential features and different  visual displays of what agents and supervisors would see when using the tool.  We got feedback from  several focus groups and used their input to make decisions on specific design elements.  Our second  step was to develop a working prototype so we could do a show-and-tell, rather than a PowerPoint  presentation, to get leadership and governance approvals.  The last step was to develop a working  solution that could be used by up to 10% of our agents simultaneously.  Because of our decision to not  pursue the more difficult A-B test, we completed the entire development cycle, from concept to learning  launch, in only 8 weeks – this was virtually unheard of and was held up as a beacon achievement.

Insights

Through our interviews with agents, we learned that the visualization tool was only marginally helpful.   It turns out that humans are pretty good at inferring customer emotion even if they are non-native  speakers, working in a written channel, and handling 2-3 conversations at once.  This was disconfirming  evidence to our main hypothesis.  However, the tool was extremely helpful for supervisors whose job is  to coach.  It allowed supervisors to monitor all interactions in real-time and intervene on at-risk  interactions while they could still make a difference.  And rather than agents feeling like their  supervisors were looking over their shoulders, agents appreciated supervisors’ pro-active intervention.   This learning made us change from thinking as agents as the primary users to thinking of supervisors as  the primary users.

Approach

I believe there were several key things we did correctly in this learning launch.  First, the development of  a paper prototype was instrumental in getting early user feedback and for keeping the software  developers focused only on the essential features – not whistles-and-bells.  Second, we accepted a  lower fidelity prototype that what would have been needed for an A-B test, and thereby obtained  course-altering learnings significantly quicker and at lower cost.

I believe the project could have gone even more smoothly if, at the beginning, we trained the project  team and key stakeholders on the design thinking process (e.g.  What is? What if? What wows? What  works?).  I found that it was very difficult for people to stay in the uncertainty of “what if” and “what  wows”.  They always wanted to jump to what works.   I believe deeper knowledge of the process may  have enabled people be more patient with the process.  Also, I believe this project was more of the  traditional R&D-to-application approach.  We discovered a technology (NLP / predictive modelling) and  applied it to customer service operations.  I think it would have been interesting to do ethnographic  interviews with agents before having any solution at all in mind, because I think we might have  discovered an unstated need that would have taken us in a totally different direction.

