Thoughts from GigaOm's Structure:Data Conference (from MIT Entrepreneurship Review)

The following article was first published in the MIT Entrepreneurship Review on April 1, 2012.

Economic expansion works like this: One trigger innovation spawns a number of supporting industries. The automobile, for example, brought us service stations, fast food chains, asphalt and suburban homes. Steel is another great example; the sturdy and affordable metal birthed the age of modern infrastructure.

Today, the world of web and mobile services has spawned a new and booming industry. The industry is called “Big Data,” and last week the key players got together in New York City at the GigaOM Structure:Data Conference.

Big Data is the result of the Internet and the decline in the cost of storage. To illustrate, just think about how much data you created and stored today. If you’re like me, you shot off a few emails, liked a Facebook comment, retweeted a Tweet and searched around for cheap plane tickets -- then ate breakfast. And this is only a sliver of my day’s data history. Swiping my debit card, getting surveillance-video taped on the subway platform and key-carding into my office created additional data points.

In short, society is creating an exponential amount of data; 1,750 exabytes in 2011, according to the Economist. The most referenced fact at the conference was that more data was created in the last two years than in all of history. And it isn’t slowing down. The growth of the Internet and the increased adoption of sensors in everything from cars to microwaves will create ever-growing amounts of data. Data is “constant and relentless,” said Comscore CTO Mike Brown.

The Business of Big Data
Big Data is the business of making sense of these disparate data points. The first and most obvious use is commercial, more specifically marketing. Nirvana in the ad world is tying your likes, tweets, check-ins, credit card transactions, and even your eye patterns when you look at a billboard to deliver the most relevant advertisement or product recommendation.

Big data goes well beyond commercial use, though. Epidemiology and Big Data are a natural fit. Disease-chasers can identify pockets of illness by tying together Facebook comments, Google searches for doctors, phone calls to doctors offices, and retail store transactions for cough syrup. And of course security is big on Big Data. The conference hosted James Woolsey, the former CIA director, who spoke about using Big Data to identify security threats to the US electric grid.

Key Trends: Real-time analytics and democratization of Big Data
There were two main themes from the conference. The first is the quest for real-time analytics. For the most part, analysis of Big Data is done in batches. Take a large retailer, for example. It may update its  predictive models with end-of-day or maybe even end-of-month SKU data. Batch updating is adequate at best, since a lot can change in a week’s time. Thus, all of the vendors at the conference pitched existing or soon-to-be-released products that enabled real-time predictive analytics. That means that once you scan that carton of milk and pack of razor blades the store’s model is updated and refined.

The second trend is the democratization of Big Data. For the last several years, Big Data has been a luxury only the top technology firms could afford. If you weren’t Google or Facebook or Amazon you simply could not attract and retain the world’s top data scientists. That is changing, though, thanks to vendors offering out-of-the-box Big Data solutions. Skytree, for example, has brought to market a solution that enables companies that aren't tech-centric to tap into their data. Skytree sits between the data, which can be stored in a relational database or a Hadoop cluster, and a front-end product like R or Matlab. Essentially, Skytree pulls in the data, looks for patterns and insight, and then pushes the data to front-end tool that most analysts are familiar with. In the coming months, the company will be releasing real-time capabilities.

What’s Next?
Big Data is slowly seeping into the mainstream conversation. Sometimes it creates outcries; for example,  The NY Times Magazine story about Target using customer data to identify pregnant woman in order to send them coupons when their “brand loyalties are up for grabs.” Other times, Big Data has been rejoiced. CellTel from Africa, for example, predicted the location of massacres in the Congo based on pre-paid phone card sales. And now the US government is fully supporting Big Data. On March 29th, the White House announced $200 million of funding to “greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data.”

In short, Big Data is a great opportunity. From a societal perspective, I’m reminded of the Saturday morning NBC public service announcement: The More You Know. We have all of this data. Now we just need to extract insights from it. From a business perspective, Big Data is booming. The culmination of ever expanding data points and practical analysis of that data equates to billions of dollars of value created for business, governments and people.