IT Infrastructure Reﬂection - Learning Launch

Challenge

Modern day IT infrastructure is a nightmare, especially at the scale of an international bank.  Imagine thousands of devices: servers, switches, routers, load balancers, proxies, and ﬁrewalls to  name a few. Now imagine these devices are running diﬀerent software and applications, and they  all need to be managed diﬀerently in terms of security, reliability, usability, scalability, defects, and  so on. You’ll be lucky if you can manage this with a thousand staﬀ members. But you can’t hire a  thousand staﬀ members. So for the staﬀ you do have, it’s critical that their time is spent eﬃciently.

That means simplifying processes and automating manual tasks wherever possible. But saving  time and money requires more than software automation. It requires understanding how the  business is operating, and knowledge of the most pressing challenges faced by members of the  infrastructure team. And since it’s nearly impossible for a small group of people to comprehend all  of the moving pieces, you need methods to help you have more success with limited information.

Selection

Solving big problems with limited information is where design thinking shines. Our team has a lot  of big ideas, but making big changes to infrastructure management is time consuming and costly.  We needed some way to test ideas quickly, and fail cheaply. This is why we chose the learning  launch. The learning launch encourages us to develop a hypothesis, design a small experimental  product that can be built and executed quickly, and learn important truths from the feedback we  receive.

Application

One complaint that we noticed over and over again was that our inventory databases were  inaccurate and out of date. In fact, we have many diﬀerent types of inventory databases. Some of  them are keeping track of our devices from a ﬁnancial accounting perspective, some are tracking  where the devices are located in a datacenter, some are monitoring if devices are performing well,  and so on. Over time, the devices can move, be replaced, install a new software version, or  experience other changes. Inevitably, errors are made, and the referential databases are not kept  up to date. You may not even realize there is a problem until you’ve wasted time looking for a  device in a location that no longer exists, or a device that wasn’t being monitored suddenly  crashes, resulting in an outage and potential ﬁnancial loss.

We suspected we could save people a lot of time and frustration by making our data problems  more visible. To test this, we designed an experiment. In two weeks, we developed a software tool  that retrieved data from two of these referential databases, and compared them for  inconsistencies. The results of the comparison were displayed in a very minimal website interface.  To gather feedback, we gave the prototype to a few diﬀerent teams who were using these  databases.

The feedback from teams was enormously positive. The web interface made it glaringly obvious  how much inaccurate data there was. We received many comments such as “I can’t believe we  didn’t have this sooner,” and “please add another database to compare.” More importantly, we  received advice about how to improve the tool very early on, such as a feature to automatically  update the databases after choosing which datasource contained the correct information.

Insight

The project is still in early stages, and the true savings of time and money may not be apparent for  months. Some cost savings can never be known. For example, we won’t know how many  incidents we prevented by having accurate data. But some savings are very tangible. For

example, some employees complained they regularly spent hours per month looking for and  reporting data errors, a task which could now be reduced to minutes.

There were several more qualitative beneﬁts. The experience gave all teams involved a better  understanding of what can be accomplished with software automation, which will help to  generate better ideas in the future. We also gained a deeper understanding of the larger data  problems that would not have been visible without this prototype to start the conversation.  Additionally, it has started a much broader reﬂection about how we can better quantify the  success of these experiments, and how we can create useful metrics to more accurately  understand the state of “what is.”

Approach

In traditional software development, a project is broken into several distinct phases, such as  requirements gathering, design, implementation, release, and maintenance. The entire process  could take years to complete. This approach is good for solving large, well-understood problems  which require careful architectural design and implementation. But this approach is not suitable  when solving dynamic business problems where much information is not known. Changing  direction during this traditional model is time consuming, and the cost of failure is high. This is  why companies are increasingly turning to solutions for faster iteration like the design thinking  learning launch, or Agile software methodologies.

