The questions in Big Data

In my last post that touched upon Big Data, I had mentioned how the seeming intent of Big Data is to synthesise actionable insights from processed and unprocessed information at touch points related or unrelated to the enterprise, and then use it to target consumers better. While this is probably true for the short-medium term, I read a wonderful perspective at GigaOm by Beau Cronin on its true potential – the path to building the equivalent of global-scale nervous systems. As I tweeted after I read it, it reminded me of something I’d written a couple of years back before I’d heard of #BigData – if we could actually use data to go beyond that to answer life’s profound questions. Before we go into the subject, here’s a nice video by OgilvyOne titled “Big Data for smarter customer experiences” (via) though it’s skewed more towards the experience rather than the data.

Beau Cronin has mentioned several possibilities this would give rise to, and the post made me think if something like the hive mind concept would mesh into it as well – a sort of hybrid neural network. He has also pointed out the hurdles we would face while we get there – gathering, processing and conversion into actionable insights, and how phenomena such as priming,expectations, and framing matter so much in how we perceive our physical and social environments. In essence, a fascinating read.

I was particularly intrigued by framing, and began thinking about it in the context of the unstructured data – tweets, posts, mails, videos – that is a major component of Big Data. The fundamental question being – is it unstructured because we’re framing it ‘wrong’? Based on the enterprise’ intent and not the users’? Ironically, I couldn’t frame the questions right until I met the ever-brilliant S, who has always maintained that the answer is easy to find once the question has been framed right. He has developed (Tulpa -to build or construct in Tibetan – is the concept he enlightened me on while we were discussing semantics) something that at a rough level mashes the MECE principle with Frame Semantics and the entity-relationship model. There’s IPR involved, so no more beans shall be spilled, but as always, I learned much from the conversation.

In essence, structure can definitely be derived from what we currently call unstructured data, provided we frame the queries right. I can intuitively begin to understand that in the era of data abundance, the only way we can make sense of all of it is by focusing on an intent that is derived from a common purpose, so that the sources of data (users) will be more open to help solve the challenges of data collection. The processing and inferences that follow would yield the best results when the right questions are asked. I have a feeling that the questions asked by a business in an earlier era might not cut it.

until next time, role models