Mike Barlow – O'Reilly Radar http://radar.oreilly.com Insight, analysis, and research about emerging technologies Wed, 08 Feb 2017 12:45:04 +0000 en-US hourly 1 https://wordpress.org/?v=4.7.5 128845193 At Solid San Francisco, you could see all the way to the future http://radar.oreilly.com/2015/07/at-solid-san-francisco-you-could-see-all-the-way-to-the-future.html http://radar.oreilly.com/2015/07/at-solid-san-francisco-you-could-see-all-the-way-to-the-future.html#respond Fri, 10 Jul 2015 16:40:29 +0000 http://radar.oreilly.com/?p=78616

Register for Solid Amsterdam, October 28, 2015 — space is limited.

I don’t usually describe conferences as “mind expanding,” but in this instance, the description is justified. At O’Reilly’s Solid 2015: Hardware, Software & the Internet of Things, held recently at Fort Mason in San Francisco, I encountered startling views of the future, thoughtful presentations on creative combinations of exciting new technologies, and warnings about what might happen if somehow it all goes wrong.

Dozens of speakers and presenters covered topics ranging from synthetic biology to augmented reality helmets to 3D printed automobiles. The audience was a mix of software developers, hardware designers, traditional manufacturers, digital manufacturers, academics, entrepreneurs, and venture capitalists keen to spot the next major tech trends.

The conference was organized around multiple tracks and themes, including data, design, software, hardware, product development, manufacturing, biology, security, technology, and startups. What follows is an overview of my key takeaways.

The big picture: Focus on interdisciplinary learning

If I had to summarize the conference in one sentence, it would be this: For the foreseeable future, progress will depend on interdisciplinary research and learning. The days of separate fiefdoms and zealously guarded areas of technical expertise are over. If we want to reap the fruits of the nascent Internet of Things (IoT) economy, we’ll need to get serious about collaborating and sharing knowledge.

  • Conference co-chairs Jon Bruner (O’Reilly Media) and Joichi Ito (MIT Media Lab) painted the big picture and detailed the profound implications of combining software, hardware, manufacturing, and biology at industrial scale. Printing genes onto silicon wafers, creating biological computers, and using cells to manufacture new products will become routine.
  • Astro Teller, leader of GoogleX, walked us carefully through the painstaking process of developing Project Loon, a global network of high-altitude balloons designed to provide Internet connectivity to the world’s most remote locations. He advised us to focus on learning, not on short-term progress. Don’t plan for success, he said, plan to learn.
  • Researcher and entrepreneur David Rose (Ditto/MIT Media Lab) showed how ubiquitous cameras are reshaping culture, communications, and commerce, creating new disciplines such as digital ethnography and spawning niche industries providing vast libraries of surrogate experiences.

Design: Don’t ask people to do too much

It’s easy to feel overwhelmed and vaguely dehumanized by a lot of the new technology emerging around us. Product developers need to temper their enthusiasm for novelty with empathy and a real understanding of how people relate to new technology.

  • Designer and author Robert Brunner (Ammunition) taught us four incredibly useful rules for designing great products for the digital era: focus on purity of purpose; make it better; don’t ask the people to do too much; and integrate design and data early on, not as an afterthought.
  • Designer Marcelo Coelho (Marcelo Coelho Studio) revealed the fascinating development of the Budweiser “Buddy Cup,” arguably the world’s first socially connected beer cup, and described his collaboration with David Cranor (Solid,Formlabs), Mengmeng Chen (Seeed), and Will Walker (Formlabs), in creating a pop-up factory in the exhibit hall at Solid to produce wearable technology for social networking.

Manufacturing and product development: Interoperability = Value of IoT

The IoT economy is real, and it’s growing fast. Ignoring the potential of the IoT isn’t merely counterproductive; it could be an extremely poor business decision. The IoT will draw resources from many sectors of the existing economy, which means there will be opportunities for practically everyone. Will your products and services fit in? Will they “work and play” well with emerging IoT environments?

  • Michael Chui (McKinsey Global Institute) reminded us why the Internet of Things is more than just another buzz phrase: McKinsey expects the IoT to generate between $3.9 and $11.1 trillion in economic value annually by 2025. The IoT’s economic potential might be higher than the hype suggests, he said, adding, that it’s actually a much bigger phenomenon than most people think it is. Interoperability of diverse systems will be the key for unlocking the IoT’s value, Chui said.
  • Investor Rob Coneybeer (Shasta Ventures) reflected on planned obsolescence as a strategy. Planned obsolescence killed Detroit, he said, but it’s more complicated in the modern age of cloud computing, digital manufacturing and overnight global competition. Despite the shades of gray, it’s always a bad idea to screw your customers, Coneybeer said.
  • Educator, entrepreneur, and “engineer of the future” Kipp Bradford (Kippworks) shared the concept of “frictionless frameworks” for accelerating innovative product development, generating value from new products more quickly and staying focused on creativity. Predictive is okay, but prescriptive is the real goal, Kipp said.
  • Product designer and investor Ben Einstein (Bolt) spelled out the complexity of developing, manufacturing, and marketing new products. Top takeaways from Einstein: If you’re planning to make your product in China, remember that most companies will want 50% of the cost paid upfront; don’t forget to factor in the scrap rate; shipping by boat is slow, but shipping by plane is really expensive; duties and custom fees can add 5% to your costs; you’ll probably need to buy liability insurance; retailers say they will pay you in 90 days, but 150 days is more common; and remember to price your product at least 2.5 times the cost of materials required to make it.
  • Inventor, designer, entrepreneur, and author Mickey McManus (Autodesk, Maya Design) described how digital manufacturing, machine learning, and the IoT have combined to create a new “primordial soup” that is dramatically and fundamentally altering human existence, for better and for worse. Moore’s Law, he said, means that any technology that’s in your car today will be in your shoes tomorrow.

Bio: The new digital

The walls between “wet” biology and “dry” technology have fallen, and a new field of digital biology is transforming the relationship between the living world and the worlds of software, hardware, and manufacturing. This is a truly revolutionary trend that will fundamentally change all of our lives.

  • Synthetic biologist, artist and community organizer David Sun Kong (MIT) explained why “biology is the new digital,” and how biological processing units (BPUs) can produce molecules on demand for performing a wide range of useful activities such as targeting and killing cancer cells. If you think the idea of fecal transplants is weird, he explained why it just might be the next important wave in health care and why researchers are working to make 3D artificial intestines to manufacture a better breed of gut microbes.
  • Chemist and researcher Gwendolyn Graff (Wrigley) took us on a journey of flavor, demonstrating how the flavors we experience are actually combinations of taste, aroma, color, texture, and reactions from our trigeminal nerve. In addition to explaining why we’re all programmed to like sweet and salty flavors, she shared the secrets of creating spearmint-flavored chewing gum.

Securing the IoT: Mad Max vs Star Trek

Despite the conference’s light-hearted atmosphere, there was an undercurrent of concern and apprehension over where all of the cool new technology is taking us. Not everything we think up should necessarily be turned into a product; there’s a whole universe of unintended consequences that should be considered first. We’re at an inflection point in our history as a culture, and the choices we make today will likely echo for decades ahead.

  • Author Cory Doctorow (Electronic Frontier Foundation) warned of the dangers of the Digital Millennium Copyright Act of 1998, which created a perfect storm for “crimeware, spyware, and malware” to run amok in “privileged environments” and severely restricts our ability to protect our devices from spying on us. “We’ve reached a level of peak indifference to surveillance,” Doctorow said, predicting the emergence of “a new kind of tenant farming for everyone” in which everyone is constantly watched and every device we “own” can be controlled remotely.
  • Cyber security expert Hugo Fiennes (Electric Imp) delivered the bad news about the brave new world we’re creating: no application is safe, every connected device is a node that can be hacked, one hacked device can cause major problems in a network, and hackers are only beginning to focus their attention on the IoT.
  • Techno-archaeologist Dennis Wingo (Skycorp) shared the wonderful story of saving irreplaceable data and restoring images of the moon’s landscape from NASA’s Apollo space missions. “We face a choice between Mad Max and Star Trek,” Wingo said, urging us to choose the latter as our blueprint for the future.
  • Programmer and game designer Henning Diedrich (IBM) explained the significance of IBM’s ADEPT project and walked us through the complex technical, legal, and social challenges of designing “trust-less” networks in a world where every device “phones home” and can be controlled remotely.

Those were only a handful of the many speakers and presenters. You can learn more about the conference and see the complete schedule of presentations here; many of the keynote speakers are also available on the Solid Conference (2015) San Francisco playlist on O’Reilly’s YouTube channel.

I came away from Solid 2015 with a much stronger grasp on the challenges and opportunities emerging from the intersection of hardware, software, biology, manufacturing, and the IoT. It’s not just hype; it’s real, and it’s already influencing many of our most basic perspectives on technology. The O’Reilly Solid conference moves to Amsterdam on October 28, and will return to San Francisco April 20-22, 2016.

Public domain image on article and category pages via The Internet Archive on Flickr.

]]>
http://radar.oreilly.com/2015/07/at-solid-san-francisco-you-could-see-all-the-way-to-the-future.html/feed 0 78616
Building functional teams for the IoT economy http://radar.oreilly.com/2015/06/building-functional-teams-for-the-iot-economy.html http://radar.oreilly.com/2015/06/building-functional-teams-for-the-iot-economy.html#respond Thu, 04 Jun 2015 15:04:47 +0000 http://radar.oreilly.com/?p=77418 Download a free copy of our new report, “When Worlds Collide: Hardware, Software, and Manufacturing Teams for the IoT,” by Mike Barlow. Editor’s note: this post is an excerpt from the report.

You don’t have to be a hardcore dystopian to imagine the problems that can unfold when worlds of software collide with worlds of hardware.

When you consider the emerging economics of the Internet of Things (IoT), the challenges grow exponentially, and the complexities are daunting.

For many people, the IoT suggests a marriage of software and hardware. But the economics of the IoT involves more than a simple binary coupling. In addition to software development and hardware design, a viable end-to-end IoT process would include prototyping, sourcing raw materials, manufacturing, operations, marketing, sales, distribution, customer support, finance, legal, and HR.

It’s reasonable to assume that an IoT process would be part of a larger commercial business strategy that also involves multiple sales and distribution channels (e.g., wholesale, retail, value-added reseller, direct); warehousing; operations; and interactions with a host of regulatory agencies at various levels of government, depending on the city, state, region, and country.

A diagram of a truly functional IoT would look less like a traditional linear supply chain and more like an ecosystem or a network. The most basic supply chain looks something like this:

Supplier → Producer → Consumer

If we took a similarly linear approach and mapped it to a hypothetical IoT scenario, it might look something like this:

Entrepreneurs → Capital Markets and Investors → Sales → Software Development → Hardware Design → Prototyping → Manufacture → Warehousing → Distribution/Logistics → Wholesale → Retail → Consumer Markets → Customer Service → Finance → Legal → HR → Regulators

Even an abbreviated description of a hypothetical IoT “supply chain” reveals the futility of attempting to map a simple linear sequential model onto a multi-dimensional framework of interconnected high-velocity processes requiring near-real-time feedback. A practical IoT ecosystem model would look something like this:

Sketch of a hypothetical IoT ecosystem.

In all likelihood, the IoT economy will resemble a network of interrelated functions, responsibilities, and stakeholders. The sketch only gives a rough idea of the complexities involved — each node would have its own galaxy of interconnected processes, technologies, organizations, and stakeholders. In any case, it’s a far cry from simple “upstream/downstream” charts that make supply chains seem like smoothly flowing rivers of pure commercial activity.

Unlike traditional software development scenarios, IoT projects are undertaken with the “real world” clearly in mind. It’s understood that IoT services and products will “touch” the physical universe we actually live in. If the assignment is developing new parts for jet engines used by passenger airlines, there will be real consequences if the parts don’t work as promised.

A more fluid approach to team building

Working relationships between software developers, hardware designers, and manufacturers within IoT ecosystems are still taking shape and evolving. The structure, staffing, and organization of a team would depend largely on the project at hand and the depth of available talent.

“There isn’t an ideal team. You have to start with the problem and work backward to discover what is available and commoditized, and what really needs to be created to make a solution real,” says Andrew Clay Shafer, senior director of technology at Pivotal, a company that provides a cloud platform for IoT developers. “The dream team would have unlimited resources, expertise, and imagination. Realistically, you won’t have a dream team, so you are better off examining the problem, looking at your available resources, and filling in gaps.”

Mike Estee, CTO and co-founder of Other Machine Co., a company that builds machines and creates software for desktop manufacturing, says the firm looks for software developers with experience in “building consumer-facing applications that integrate directly with hardware designed in tandem” and hardware designers who can bring products to market “on time, on budget, and with a high degree of polish.”

Ayah Bdeir is the founder and CEO of littleBits, a company that creates “libraries” of electronic modules that snap together with magnets. An advocate of the open source hardware movement, she is a co-founder of the Open Hardware Summit, a TED Senior Fellow, and an alumna of the MIT Media Lab. From her perspective, the “ideal” development team includes hardware, firmware, software, and Web experts. Tightly integrating the design function is absolutely critical, she says.

“Design is crucial at the very beginning. Our second hire in the company was an industrial designer, and it has transformed our business and our product,” Bdeir says. “Specific skills and languages don’t always mean as much as the ability to adapt. The team needs to be creative, agile, and quick to learn.”

While software development is considered a relatively “mature” field, the need to combine software development and hardware design has spawned a host of new problems. In an email, Bdeir shared her view of the current situation:

For software developers, a lot of the barriers have been removed. Today, you can prototype and launch a software product with very little time, money, and most importantly, few external dependencies. When you see there is traction, you can scale your offering gradually, and not overextend yourself financially.

For hardware, the barriers are much more defined and difficult to deal with. You can prototype a lot faster, and get initial interest/traction through Kickstarter, press, or showing off your product at events. But there is a pretty big step to getting first-run manufacturing done.

Finding manufacturing partners who will help you make your product manufacturable is hard when you are at a small scale, and it takes a lot of time. When we were starting out, we had to manufacture our magnetic connector with a factory that made plastic insoles and knee guards because I couldn’t find any manufacturer that knew electronics or connectors that would take me seriously.

Estee, who spent nine years as a software engineer at Apple, has a similarly chastened view of the challenges. Here’s a brief excerpt from an essay he posted recently on Medium:

If you’ve never started a hardware company before, it’s very hard to understand the vast differences between a prototype, a product, and a company that supports it. A product is so much more than just the machine. It’s also the packaging, the quality assurance plan, the documentation, the factory, the supply chain, vendor relations, testing, certification, distribution, sales, accounting, human resources, and a health plan. It goes on and on. All things you probably weren’t thinking about in the rush to get the Kickstarter launched.

Indeed, the complexities seem nearly inexhaustible. Shafer describes the challenges from an industry-wide perspective:

As we move into a world where all things are integrated, the stack is getting wider and deeper. The choice is between building teams that have expertise at each level … or choosing platforms to do the heavy lifting and act as force multipliers for teams with more focused skills. The challenge is deciding which problems to solve yourself and which to outsource.

People get excited about creating ‘things,’ but the Internet is the other half of the equation, and there are still hard problems to solve there. Even moderate IoT build-outs could quickly create sensor networks that have latency, scaling, and analytics problems that only a handful of the big Web companies have had to solve until now.

Today, there is no equivalent to the LAMP1 stack for the IoT. It will be several years before a common way of solving IoT problems begins to emerge.

]]>
http://radar.oreilly.com/2015/06/building-functional-teams-for-the-iot-economy.html/feed 0 77418
Big data’s move to the cloud http://radar.oreilly.com/2014/10/big-datas-move-to-the-cloud.html http://radar.oreilly.com/2014/10/big-datas-move-to-the-cloud.html#comments Wed, 08 Oct 2014 11:00:55 +0000 http://radar.oreilly.com/?p=71147 One night when our son was two years old, he abruptly decided that he didn’t like taking baths. As my wife recalls, he struggled mightily against the ritual of bathing for several months until, suddenly and mysteriously, he decided that he liked bathing again. We’re happy to report that he has managed to stay relatively clean ever since.

When I speak with CIOs and other IT leaders about moving big data operations into the cloud, I am reminded of our son’s unexplained loathing of the bathtub.

Nearly everyone associated with IT understands that most IT operations — including big data analytics — must eventually move into the cloud. The traditional on-premises approaches are simply too costly, and CIOs are under crushing pressure to shift budgetary resources to value-added, customer-facing activities.

For most companies, the writing is already on the wall. The cloud offers greater agility and elasticity, and quicker product development cycles — and can reduce costs. When you add up the benefits, it seems inevitable that the bulk of IT operations will move into the cloud. Nevertheless, the foot-dragging and excuse-making continues.

The common reasons for avoiding cloud-based big data services include security, performance, lack of bandwidth, and loss of data accuracy. The cloud vendors I speak with are well aware of those obstacles, and they’re working hard to overcome them.

From my perspective as a writer who covers IT management trends, the reluctance to embrace the cloud seems to be based more on inertia than objective reasoning. For many CIOs, it’s easier to stick with the status quo — even if the status quo means spending more money than necessary.

It’s understandable that CIOs aren’t racing into the cloud. Nobody wants to be the first one into the pool. But my hunch is that the tipping point will arrive fairly soon, and there will be a sudden rush of companies looking to ink deals with cloud-based big data providers.

My hunch is supported by a recent survey of software developers, systems architects, engineers, data scientists, and data analysts conducted by John King, a data analyst at O’Reilly (you can read my report on the survey here). In the survey, King discovered that after a company has gained some experience using big data in the cloud, it is more likely to increase its use of similar big data services.

The survey also shows that roughly 40% of respondents who identify themselves as big data practitioners currently use cloud services for analytics and that roughly 55% of respondents who plan to become big data practitioners also plan to use cloud services for analytics.

Those findings indicate the market is ready for cloud-based big data services. I can’t predict exactly when everyone will start jumping into the pool, but it’s likely to happen sooner rather than later as CIOs race to shift limited IT resources from inward-facing operations to value-added, customer-facing activities.

CIOs and IT vendors have been talking for years about the importance of moving into the cloud. I’m guessing that within the next six months, the talk will turn to action and we’ll be wondering why everyone put up such a fuss for such a long time.

For more details on the survey results, download the free report, Migrating Big Data Analytics into the Cloud — compliments of Teradata.

This post is part of a collaboration between O’Reilly and Teradata exploring the use of cloud-based big data tools. See our statement of editorial independence.

]]>
http://radar.oreilly.com/2014/10/big-datas-move-to-the-cloud.html/feed 1 71147
Roll-your-own database architecture http://radar.oreilly.com/2014/06/roll-your-own-database-architecture.html http://radar.oreilly.com/2014/06/roll-your-own-database-architecture.html#respond Tue, 24 Jun 2014 14:00:27 +0000 http://radar.oreilly.com/?p=68788

Two years ago, most of the conversations around big data had a futuristic, theoretical vibe. That vibe has been replaced with a gritty sense of practically. Today, when big data or some surrogate term arises in conversation, the talk is likely to focus not on “what if,” but on “how do we get it done?” and “what will it cost?”

Real-time big data analytics and the increasing need for applications capable of handling mixed read/write workloads — as well as transactions and analytics on “hot” data — are putting new pressures on traditional data management architectures.

What’s driving the need for change? There are several factors, including a new class of apps for personalizing the Internet, serving dynamic content, and creating rich user experiences. These apps are data driven, which means they essentially feed on deep data analytics. You’ll need a steady supply of activity history, insights, and transactions, plus the ability to combine historical analytics with hot analytics and read/write transactions.

Some people think all the data should sit in one big Hadoop data lake; others think that all data should sit in memory for speed. Here’s another potential scenario: store historical data on cost-effective rotational media, while storing more valuable, recent data on solid-state drives (SSD). That kind of combined architecture would enable you to to run historical analytics on the older data and hot analytics and transactions on the newer and more valuable data.

It’s no secret that VCs are pouring cash into startups promising a new generation of advanced analytics. The rapidly growing supply — some might say oversupply — of newer analytics will drive even greater adoption and usage, highlighting (and in many instances creating) processing bottlenecks that simply did not exist two years ago. In other words, scale is now a serious issue.

Processing bottlenecks are painful for users, but they create wonderful opportunities for innovators who are willing to explore new database management architectures. This seems like a perfect moment for a new wave of “roll your own” configurations designed specifically to handle unique and unorthodox amalgams of data.

That said, there’s also a window for vendors who can devise smart, powerful, flexible data management architectures that will take you where you need to go without breaking the bank.

Whether you’re doing it yourself or relying on a vendor, blended architectures should let you “scale out” before you scale up, which spares you the embarrassment of flying blindly into terra incognita, waving your checkbook and looking for a consultant to rescue you.

Sooner rather than later, the larger issue of database architecture evolution must be addressed. As we know, evolution doesn’t proceed according to a plan. It moves in fits and starts. At this moment, it seems manifestly apparent that traditional database architectures are no longer sufficient. What’s not entirely clear, however, is the logical next step.

If your business requires high transactional performance on multi-terabyte datasets, and you’re concerned about how many servers you’re running, a blended architecture might be a step in the right direction.

This post is part of a collaboration between O’Reilly and Aerospike. See our statement of editorial independence.

Photo by Kenny Louie, used under a Creative Commons license.

]]>
http://radar.oreilly.com/2014/06/roll-your-own-database-architecture.html/feed 0 68788
Extracting value from the IoT http://radar.oreilly.com/2014/06/extracting-value-from-the-iot.html http://radar.oreilly.com/2014/06/extracting-value-from-the-iot.html#comments Mon, 23 Jun 2014 11:00:34 +0000 http://radar.oreilly.com/?p=68733

Union Pacific uses infrared and audio sensors placed on its tracks to gauge the state of wheels and bearings as the trains pass by.

The Internet of Things (IoT) is more than a network of smart toasters, refrigerators, and thermostats. For the moment, though, domestic appliances are the most visible aspect of the IoT. But they represent merely the tip of a very large and mostly invisible iceberg.

IDC predicts by the end of 2020, the IoT will encompass 212 billion “things,” including hardware we tend not to think about: compressors, pumps, generators, turbines, blowers, rotary kilns, oil-drilling equipment, conveyer belts, diesel locomotives, and medical imaging scanners, to name a few. Sensors embedded in such machines and devices use the IoT to transmit data on such metrics as vibration, temperature, humidity, wind speed, location, fuel consumption, radiation levels, and hundreds of other variables.

“Machines can be very chatty,” says William Ruh, a vice president and corporate officer at GE.

Ruh’s current focus is to drive the company’s efforts to develop an “industrial” Internet that blends three elements: intelligent machines, advanced analytics, and empowered users. Together, those elements generate a variety of data at a rapid pace, creating a deluge that makes early definitions of big data seem wildly understated.

Making sense of that data and using it to produce a steady stream of usable insights require infrastructure and processes that are fast, accurate, reliable, and scalable. Merely collecting data and loading it into a data warehouse is not sufficient — you also need capabilities for accessing, modeling, and analyzing your data; a system for sharing results across a network of stakeholders; and a culture that supports and encourages real-time collaboration.

What you don’t need is a patchwork of independent data silos in which information is stockpiled like tons of surplus grain. What you do need are industrial-grade, integrated processes for managing and extracting value from IoT data and traditional sources.

Dan Graham, general manager for enterprise systems at Teradata, sees two distinct areas in which integrated data will create significant business value: product development and product deployment.

“In the R&D or development phase, you will use integrated data to see how all the moving parts will work together and how they interact. You can see where the friction exists. You’re not looking at parts in isolation. You can see the parts within the context of your supply chain, inventory, sales, market demand, channel partners, and many other factors,” says Graham.

The second phase is post-sales deployment. “Now you use your integrated data for condition-based (predictive) maintenance. Airplanes, locomotives, earth movers, automobiles, disk drives, ATMs, and cash registers require continual care and support. Parts wear out and fail. It’s good to know which parts from which vendors fail, how often they fail, and the conditions in which they fail. Then you can take the device or machine offline and repair it before it breaks down,” says Graham.

For example, microscopic changes in the circumference of a wheel or too little grease on the axle of a railroad car, can result in delays and even derailments of high-speed freight trains. Union Pacific, the largest railroad company in the US, uses a sophisticated system of sensors and analytics to predict when critical parts are likely to fail, enabling maintenance crews to fix problems while rolling stock is in the rail yard. The alternative, which is both dangerous and expensive, would be waiting for parts to fail while the trains are running.

Collecting data and loading it into a data warehouse is not sufficient. You also need capabilities for accessing, modeling, and analyzing your data.Union Pacific uses infrared and audio sensors placed on its tracks to gauge the state of wheels and bearings as the trains pass by. It also uses ultrasound to spot flaws or damage in critical components that could lead to problems. On an average day, the railroad collects 20 million sensor readings from 3,350 trains and 32,000 miles of track. It then uses pattern-matching algorithms to detect potential issues and flag them for action. The effort is already paying off: Union Pacific has cut bearing-related derailments by 75%, says Graham.1

NCR Corporation, which pioneered the mechanical cash register in the 19th century, is currently the global leader in consumer transaction technologies. The company provides software, hardware, and services, enabling more than 485 million transactions daily at large and small organizations in retail, financial, travel, hospitality, telecom, and technology sectors. NCR gathers data telemetrically from the IoT — data generated by ATMs, kiosks, point-of-sale terminals, and self-service checkout machines handling a total of about 3,500 transactions per second. NCR then applies its own custom algorithms to predict which of those devices is likely to fail and to make sure the right technician, with the right part, reaches the right location before the failure occurs.

Underneath the hood of NCR’s big data/IoT strategy is a unified data architecture that combines an integrated data warehouse, Hadoop, and the Teradata Aster Discovery Platform. The key operating principle is integration, which assures that data flowing in from the IoT is analyzed in context with data from multiple sources.

“The name of the game is exogenous data,” says Michael Minelli, an executive at MasterCard and co-author of Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today’s Businesses. “You need the capabilities and skills for combining and analyzing data from various sources that are outside the four walls of your organization. Then you need to convert data into actionable insights that will drive better decisions and grow your business. Data from the IoT is just one of many external sources you need to manage in combination with the data you already own.”

From Minelli’s perspective, data from the IoT is additive and complementary to the data in your data warehouse. Harvey Koeppel, former CIO at Citigroup Global Consumer Banking, agrees. “The reality is that there is still a legacy environment, and it’s not going away anytime soon. Facts are facts; they need to be collected, stored, organized, and maintained. That’s certainly the case for Fortune 1000 companies, and I expect it will remain that way for the foreseeable future,” says Koeppel.

Big data collected from the IoT tends to be “more ephemeral” than traditional types of data, says Koeppel. “Geospatial data gathered for a marketing campaign is different than financial data stored in your company’s book of record. Data that’s used to generate a coupon on your mobile phone is not in the same class as data you’re required to store because of a government regulation.”

That said, big data from the IoT is rapidly losing its status as a special case or oddity. With each passing day, big data is perceived as just another item on the menu. Ideally, your data architecture and data warehouse systems would enable you to work with whichever type of data you need, whenever you need it, to create actionable insights that lead to improved outcomes across a variety of possible activities.

“In the best of all worlds, we would blend data from the IoT with data in the data warehouse to create the best possible offers for consumers in real time or to let you know that your car is going to run out of gas 10 minutes from now,” says Koeppel. “The thoughtful approach is combining data from a continuum of sources, ranging from the IoT to the traditional data warehouse.”


1 Murphy, Chris. “High-Speed Analytics: Union Pacific shows the potential of the instrumented, interconnected, analytics-intensive enterprise.” Information Week, August 13, 2012.

This post is part of a collaboration between O’Reilly and Teradata exploring the convergence of hardware and software. See our statement of editorial independence.

Photo by Rick Cooper, on Wikimedia Commons.

]]>
http://radar.oreilly.com/2014/06/extracting-value-from-the-iot.html/feed 1 68733
How to be agile with your big data http://radar.oreilly.com/2014/05/how-to-be-agile-with-your-big-data.html http://radar.oreilly.com/2014/05/how-to-be-agile-with-your-big-data.html#respond Thu, 29 May 2014 11:00:08 +0000 http://radar.oreilly.com/?p=68589 Data analysis, like other pursuits, is a balancing act. The rise of big data ratchets up the pressure on the traditional enterprise data warehouse (EDW) and associated software tools to handle rapidly evolving sets of new demands posed by the business. Companies want their EDW systems to be more flexible and more user friendly — without sacrificing processing speeds, data integrity, or overall reliability.

“The more data you give the business, the more questions they will ask,” says José Carlos Eiras, who has served as CIO at Kraft Foods, Philip Morris, General Motors, and DHL. “When you have big data, you have a lot of different questions, and suddenly you need an enterprise data warehouse that is very flexible.”

EDWs are remarkably powerful, but it takes considerable expertise and creativity to modify them on the fly. Adding new capabilities to the EDW generally requires significant investments of time and money. You can develop your own tools internally or purchase them from a vendor, but either way, it’s a hard slog.

“You wind up with an endless proliferation of tools, each slightly different, designed to handle a specific type of query,” says Eiras. “Every time you have a different question, you have to use a different tool. It isn’t a good situation.”

In today’s ultra-competitive markets, business operates far too quickly for traditional software development timetables, and most companies are rightfully wary of relying on outside vendors to solve ongoing business challenges. Luckily, agile and other “lean” software development methods offer repeatable processes for harnessing creativity and expertise, at a pace that’s swift enough to keep the business happy.

Achieving concrete results faster

Unlike traditional ITIL-based software development life cycle (SDLC) or “waterfall” methods, agile does not begin with a finely detailed compendium of abstract requirements. Instead of striving for perfection, agile aims for the minimum viable product (MVP), which means that agile produces concrete results faster than traditional development methods.

The difference between the “abstractedness” of traditional methods and the “concreteness” of agile might seem like a matter of semantics, but it’s highly relevant in a world of hyper-turbulent markets and fickle customers. Abstractions are tolerable when you have plenty of time and money, but when time and money are tight, you need concrete results in a hurry. The whole point of agile is finding out quickly whether your idea works or not. Agile embodies the concept of “failing fast,” a phrase that sums up the ethos of modern innovation.

But here’s the rub: adopting agile methodologies is not the same as being agile. Agile is both a process and a mindset. It embraces both discipline and flexibility. Agile is not some kind of free-form anarchy; it’s a structured way of producing usable software without having a pre-written script, like improvisational theatre.

The first rule of improv is always to say, “Yes, and…” Everything an actor does on stage is complementary; nothing is exclusionary. New information emerges unexpectedly, and continual change is a given, but established premises remain in place.

Cultural challenges

“The biggest challenge is cultural,” says Oliver Ratzesberger, senior vice president of software at Teradata Labs. “Executives tell me they want agility, and then in the next sentence they ask me for a project plan, a roadmap, and a product release date. They switch back to waterfall without even realizing it.”

For modern software executives like Ratzesberger, explaining the difference between agile and “agility” is not an abstract dilemma. “Agility, especially in the context of big data, takes a lot of effort and really hard work. Creating an agile environment doesn’t mean turning off governance, eliminating documentation, and giving developers a sandbox to work in. That’s not agility — that’s the Wild West,” says Ratzesberger.

The problem with taking a Wild West approach to software development is that, “you end up with results that are not reproducible, which very quickly erodes confidence in your ability to handle the data,” he says. If the business executives who depend on the EDW don’t trust you with their data, they will look elsewhere for answers to their questions.

“Without methodology, agile quickly runs away on you,” says Ratzesberger. “You need to train people, and you need to change the culture of your organization. If you’re applying agile to big data, you need to work on the agile piece first, and then overlay the big data on top of it. You can’t start with big data and then try to make it agile. Then it becomes the Wild West, and you’re in trouble.”

Train the business, too

Explaining agile to people working in the business units is also important, says Ratzesberger. “You need to train the business. If you only train the IT people, the people in the business units won’t understand what you’re doing. They will assume they need to submit a requirements document, which they dread, and then there will be no agility between IT and the business.” He suggests co-locating technology and business people to improve communication and collaboration during agile projects.

Agile is not necessarily the answer to every software development challenge, says Ratzesberger. “There are certain tasks where accuracy is paramount. In production, for example, you’re not going to upgrade a critical process using agile. If it’s a customer-facing process that the company depends on for revenue, then agile might not be the best path.”

Agile is optimally suited for situations in which the ability to generate lightning-fast results can produce genuine competitive advantages. Experimenting with subsets of customer data, testing new product categories, evaluating the usability of a web page design, gauging the appeal of a smartphone app — those are the best scenarios for getting the most value from agile.

Complementary technologies

Jim Tosone, former director and team lead of the Healthcare Informatics Group at Pfizer Pharmaceuticals, uses the principles and processes of improvisation in his leadership development practice to help clients improve business performance.

Tosone sees agile, big data, and the EDW as natural partners. “Big data technology such as Hadoop, which is based on open-source code, is inherently agile,” he says.

“Now that we’re seeing more SQL-on-Hadoop integrations, we have a global community of open-source developers working to solve EDW challenges. That phenomenon is likely to accelerate,” says Tosone. “The relationships between agile, big data, and EDW are complimentary rather than antagonistic.”

Where to begin

Companies looking for ways to bring agile into their EDW operations should begin with simple steps. Tosone recommends outlining key use cases (e.g., hypothesis generation, hypothesis testing, business plan execution) and forming developer teams with diverse backgrounds. Diversity will assure that a broad range of possible solutions are considered.

Then map the potential solutions to each use case. Look for solutions that will enable users to run queries and analyses across multiple platforms. Favor solutions that can be modified, iterated, and replaced easily.

Tosone agrees with Ratzesberger that changing the culture is critical to success. “You need to educate and train people, and help them feel comfortable working across multiple functions and disciplines,” he says. “True agility requires a mindset that perceives change as a positive instead of a negative. You need a bias toward exploration and patience to resist the desire for closure.”

Hybrid environments

It seems clear that agile offers a reasonably quick and cost-effective way to bring flexibility to the EDW and to integrate newer open-source technologies, such as Hadoop, with existing database systems. Flexibility translates into greater speed and cost savings; integration promises a significantly wider range of analytic capabilities, which creates more opportunities for the business to pursue.

As we move forward into an era of bigger, faster, and more effective data analytics, it seems logical to assume that database systems architectures will include both traditional and open-source components. In hybrid computing environments, the real challenge is optimizing the relationships between all of the various people, processes, and technologies required to get the job done quickly and efficiently.

Instead of picking fights over platforms, we should search for mutually beneficial scenarios in which we use available resources to help businesses gain advantages in competitive markets. Or, as my friends in improvisational theater would say: “Yes, and…”

This article is part of a collaboration between O’Reilly and Teradata. See our statement of editorial independence.

]]>
http://radar.oreilly.com/2014/05/how-to-be-agile-with-your-big-data.html/feed 0 68589
Venal Sins: Cash, Sex, and IT Infrastructure http://radar.oreilly.com/2014/04/venal-sins-cash-sex-and-it-infrastructure.html http://radar.oreilly.com/2014/04/venal-sins-cash-sex-and-it-infrastructure.html#respond Wed, 09 Apr 2014 17:00:27 +0000 http://strata.oreilly.com/?p=62176 Yet again, I reveal the base instincts driving my interest in big data. It’s not the science – it’s the cash. And yes, on some level, I find the idea of all that cash sexy. Yes, I know it’s a failing, but I can’t help it. Maybe in my next life I’ll develop a better appreciation of the finer things, and I will begin to understand the real purpose of the universe…

Until then, however, I’m happy to write about the odd and interesting intersection of big data and big business. As noted in my newest paper, big data is driving a renaissance in IT infrastructure spending. IDC, for example, estimates that worldwide spending for infrastructure hardware alone (servers, storage, PCs, tablets, and peripherals) will rise from $461 billion in 2013 to $468 billion in 2014. Gartner predicts that total IT spending will grow 3.1% in 2014, reaching $3.8 trillion, and forecasts “consistent four to five percent annual growth through 2017.” For a lot of people, including me, the mere thought of all that additional cash makes IT infrastructure seem sexy again.

Of course, there’s more to the story than networks, servers, and storage devices. But when people ask me, “Is this big data thing real? I mean, is it real???” the easy answer is yes, it must be real because lots of companies are spending real money on it. I don’t know if that’s enough to make IT infrastructure sexy, but it sure makes it a lot more fascinating and – dare I say it, intriguing – than it seemed last year.

In life, sex is the key to survival. In business, cash is king. Is there a connection? Read my paper, and please let me know.

Get your free digital copy of Will Big Data Make IT Infrastructure Sexy Again? — compliments of Syncsort.

]]>
http://radar.oreilly.com/2014/04/venal-sins-cash-sex-and-it-infrastructure.html/feed 0 62176
CIOs and the Big Data Challenge http://radar.oreilly.com/2013/10/cios-and-the-big-data-challenge.html http://radar.oreilly.com/2013/10/cios-and-the-big-data-challenge.html#comments Mon, 14 Oct 2013 17:00:14 +0000 http://strata.oreilly.com/?p=60320 The trend is clear: The CIO’s IT budget is getting smaller and the CMO’s IT budget is getting larger. As a result, the CIO’s role is diminishing and the CMO’s role is expanding. From a business perspective, the shift feels inevitable. Despite talk about transforming corporate IT organizations from cost centers into profit centers, the role of the CIO has remained largely administrative.

In their hearts, most CIOs know the score. They’ve won their battle to earn “a seat at the table,” but the table has gotten smaller. The main challenges ahead of them are technical, not strategic. Their key areas of focus today are mobility, cloud, and security. They are aware of big data, but it’s just not a survival issue for them.

That doesn’t mean that CIOs don’t care, it just means that most of them don’t have the bandwidth to deal with the challenges of big data. The CIOs I interviewed in my newest report do not constitute a representative sample of the CIO community. My sources tend to be people who are keenly and genuinely interested in the convergence of big data, statistical analysis, behavioral science, and marketing. That makes them outliers. I love interviewing them, but I don’t believe they speak for the majority of CIOs.

That said, I think the new report will be valuable to anyone who is considering making a move from IT into marketing. The marketing team will welcome people with technical chops who also understand how business works.

I honestly don’t see the shift of dollars as a “win-lose” or “zero-sum” game. It’s all for the good. Big data is rapidly becoming the life blood of marketing, and marketing drives revenue. As Ben Bradley said, “Follow the money.”

Download The Changing Role of the CIO for free.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England
Strata Rx Health Data Conference: April 23-25, 2014 | Boston, MA

]]>
http://radar.oreilly.com/2013/10/cios-and-the-big-data-challenge.html/feed 4 62666
Big Data Culture Gap: Technology Advancing More Quickly Than People and Processes http://radar.oreilly.com/2013/09/big-data-culture-gap-technology-advancing-more-quickly-than-people-and-processes.html http://radar.oreilly.com/2013/09/big-data-culture-gap-technology-advancing-more-quickly-than-people-and-processes.html#comments Mon, 16 Sep 2013 17:00:41 +0000 http://strata.oreilly.com/?p=59859

Way back in October 2012, mere days before Hurricane Sandy filled our basement with five feet of water, the nice editors at O’Reilly Media asked me to write a white paper on the emerging architecture of big data. That paper was followed by two more, one about the emerging culture of big data, and another about the impact of big data on the traditional IT function.

Despite my attempts to make all three papers seem wildly different, they all share a common theme or subtext, namely that the technology of big data is evolving far more quickly than the people and processes of big data.

In other words, the tools are more advanced than the organizations using them. At least that’s my takeaway. After interviewing dozens of data analysts, industry experts, and senior-level corporate executives, I’m convinced that the technology is advancing faster than the abilities of the people trying to use it.

In retrospect, it’s not surprising that the technology of big data has evolved more rapidly than the organizational structures required to harness big data and convert it into a steady source of value. C-level executives and boards of directors regard big data as promising, but they weren’t born yesterday, and they need to see the business case before they start writing big checks. “Tactics in search of a strategy” is how one senior executive recently summed up his thoughts on big data.

Many of us sense that big data is destined to become the next big thing, but none of us is quite sure how it will all play out.

We can take comfort in the knowledge that this has all happened before. When airplanes were first pressed into military service, they were used exclusively for reconnaissance. When a team of engineers led by Anthony Fokker developed the synchronized machine gun, airplanes morphed into lethal weapons and a new strategy—aerial warfare—was born.

Most of us know the story of how the folks at Xerox didn’t understand the value of the clunky computer mouse they had invented, and how Steve Jobs took the basic idea and engineered it into a practical tool that helped launch a revolution in personal computing.

Based on what I’ve been hearing from my sources, I have the feeling that we are five or six inventions away from a similar revolution in big data. I’m not really sure what form those inventions will take—or I would quit my day job and invest in the companies making them—but I am certain that multiple disciplines and technologies will be involved.

When it arrives, the big data revolution will transform the modern enterprise, accelerate the growth of markets, and launch a new era of social commerce. The changes—particularly in emerging economies—will be mind-boggling in both scale and scope.

Ten years from now, we’re going to look back and wonder why we didn’t see it coming, but by then we’ll be on to the next big thing, and big data will seem about as interesting as a laptop from the 1990s. Meantime, encourage your kids to learn Python, Ruby, and R.

Cheers!

Download The Culture of Big Data for free.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

]]>
http://radar.oreilly.com/2013/09/big-data-culture-gap-technology-advancing-more-quickly-than-people-and-processes.html/feed 1 62645
Big data vs. big reality http://radar.oreilly.com/2013/06/big-data-vs-big-reality.html http://radar.oreilly.com/2013/06/big-data-vs-big-reality.html#comments Tue, 04 Jun 2013 13:00:50 +0000 http://radar.oreilly.com/?p=57418 This post originally appeared on Cumulus Partners. It’s republished with permission.

Quentin Hardy’s recent post in the Bits blog of The New York Times touched on the gap between representation and reality that is a core element of practically every human enterprise. His post is titled “Why Big Data is Not Truth,” and I recommend it for anyone who feels like joining the phony argument over whether “big data” represents reality better than traditional data.

In a nutshell, this “us” versus “them” approach is like trying to poke a fight between oil painters and water colorists. Neither oil painting nor water colors are “truth”; both are forms of representation. And here’s the important part: Representation is exactly that — a representation or interpretation of someone’s perceived reality. Pitting “big data” against traditional data is like asking you if Rembrandt is more “real” than Gainsborough. Both of them are artists and both painted representations of the world they perceived around them.

The problem with false arguments like the one posed by Hardy is that they obscure the value of data — traditional data and big data — and the impact of data on our culture. I’m now working my way through “Raw Data” is an Oxymoron, an anthology of short essays about data. I recommend it for anyone who is seriously interested in thinking about the many ways in which data has influenced (and continues influencing) our lives. I especially recommend “facts and FACTS: Abolitionists’ Database Innovations,” by Ellen Gruber Garvey. As its title suggests, the essay focuses on what proves to be an absolutely fascinating period of U.S. history in which the anti-slavery movement harvested data from real advertisements in Southern newspapers to paint a vivid and believable picture of the routine horrors inflicted by the slave system on real human beings.

That 19th century use of data mining built support for the anti-slavery movement, both in the U.S. and in England. The data played a key role in making the case for abolishing slavery — even though it required the bloodiest war in U.S. history to make abolition a fact.

Data itself has no quality. It’s what you do with it that counts.

]]>
http://radar.oreilly.com/2013/06/big-data-vs-big-reality.html/feed 5 57418
{"cc_re__cc_re__cc_by_2_0": [" our statement of editorial independence</a>.</em></p>\n<p><em>Photo by <a href=\"http://www.flickr.com/photos/kwl/2785555259/in/photolist-5f9GwP-iKwykJ-dX2KQv-78dBpB-5NByM-mcHoWr-9wK1JL-knhQaH-4dtDKE-iJYnuQ-8YhFG-6eLT7q-6MUWxV-a66NeM-9hHYKC-aggKtc-6THwj-72VCcq-k6Wyd-7qEmhn-f7ebD-kstnPJ-2QqcH1-kQCU-K7e6s-k6Wyb-nHNzSh-thsjk-2PrPUX-c43qR-aJZuLH-cuMMw3-jNpjXh-7BkMB3-dMCten-8wS8f-6RoKZm-aySPbX-6JRfPT-GW7Fm-N9oB-8119Px-kbgqBg-66hjMN-8eLn3Y-Yip8g-fPTDri-gDf5-9J5Aip-4RhJN1\">Kenny Louie</a>, used under a <a href=\"http://creativecommons.org/licenses/by/2.0/\">Creative Commons license</a>.</em></p>\n]]></content:encoded>\n\t\t\t<wfw:commentRss>http://radar.oreilly.com/2014/06/roll-your-own-database-architecture.html/feed</wfw:commentRss>\n\t\t<slash:comments>0</slash:comments>\n\t<post-id xmlns=\"com-wordpress:feed-additions:1\">68788</post-id>\t</item>\n\t\t<item>\n\t\t<title>Extracting value from the IoT</title>\n\t\t<link>http://radar.oreilly.com/2014/06/extracting-value-from-the-iot.html</link>\n\t\t<comments>http://radar.oreilly.com/2014/06/extracting-value-from-the-iot."], "copyright__copyright__copyright": [""]}