The huge growth of global data is outpacing storage

The emergence of wireless technologies, smart products and software-defined businesses is having a staggering effect on the volume of the world’s data.

Due, in part, to this Internet of Things, the digital universe is doubling in size every two years and will multiply 10-fold between 2013 and 2020 – from 4.4 trillion gigabytes to 44 trillion gigabytes.

These are the finding of a new study by EMC, a provider of IT storage hardware solutions, titled ‘The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things’.

The Internet of Things comprises billions of everyday objects that are equipped with unique identifiers and the ability to automatically record, report and receive data – a sensor in your shoe tracking how fast you run or a bridge tracking traffic patterns. According to IDC, the number of devices or things that can be connected to the Internet is approaching 200 billion today, with 7% (or 14 billion) already connected to and communicating over the Internet. The data from these connected devices represents 2% of the world’s data today. IDC now forecasts that, by 2020, the number of connected devices will grow to 32 billion – representing 10% of the world’s data.

Jeremy Burton, president of Products and Marketing, EMC Information Infrastructure, said: "As more and more businesses capitalise on the social and mobile phenomenon, the enormity and potential of the digital universe grows, and businesses are presented with greater opportunities to analyse new streams of data and gain more value from the data they already have.

"Simply put – companies of all types are shape shifting into software-defined enterprises right before our eyes. While the potential is massive, the implications are equally daunting. IT departments must press the restart button to find new ways to innovate around existing infrastructure while positioning themselves to dive into a future of third platform computing."

The Internet of Things will also influence the massive amounts of "useful data" – data that could be analysed – in the digital universe. In 2013, only 22% of the information in the digital universe was considered useful data, but less than 5% of the useful data was actually analyzed – leaving a massive amount of data lost as dark matter in the digital universe. By 2020, more than 35% of all data could be considered useful data, thanks to the growth of data from the Internet of Things, but it will be up to businesses to put this data to use.

This phenomenon will present radical new ways of interacting with customers, streamlining business cycles, and reducing operational costs, stimulating trillions of dollars in opportunity for businesses. Conversely, it presents significant challenges as businesses look manage, store and protect the sheer volume and diversity of this data. For example, IDC estimates that 40% of the data in the digital universe require some level of protection, from heightened privacy measures to fully-encrypted data. That said, only half of that data – just 20% – is actually protected.

Vernon Turner, senior VP, IDC, said: "The Digital Universe and The Internet of Things go hand in hand. As sensors become connected to the Internet, the data that they generate becomes increasingly important to every aspect of business, transforming old industries into new relevant entities. Traditional storage services will be elevated to new levels of resiliency and tolerance to support the Digital Universe, which can only be guaranteed in a software-defined environment."

Other key findings:

– Emerging markets are producing more data: Currently, 60% of data in the digital universe is attributed to mature markets such as Germany, Japan, and the United States, but by 2020, the percentage will flip, and emerging markets including Brazil, China, India, Mexico and Russia will account for the majority of data.

– Data is outpacing storage: The world’s amount of available storage capacity (i.e., unused bytes) across all media types is growing slower than the digital universe. In 2013, the available storage capacity could hold just 33% of the digital universe. By 2020, it will be able to store less than 15%. Fortunately, most of the world’s data is transient (e.g. Netflix or Hulu stream, Xbox ONE game interactions, Digital TV.) and requires no storage.

– Data touched by the cloud will double: In 2013, less than 20% of the data in the digital universe was "touched" by the cloud. By 2020, that percentage will double to 40%.

– Consumers create data but enterprises are responsible for it: Two-thirds of the digital universe bits are created or captured by consumers and workers, yet enterprises have liability or responsibility for 85% of the digital universe.