Big Data, Big Mess, Big Ideas

What's the trouble with big data? It’s obvious of course. The more data you have the more work you need to do to extract any actionable insights. Organizing and categorizing the mess seems to be the most natural first step. But what if in all the culling, parsing, simplifying, curating and aggregating you lose that hidden thread to the real insight? We offload some of this tedious work to powerful CPU’s and elegant algorithms. The more “human” part comes when we try to put the remaining, and hopefully most salient pieces of the puzzle together. If we lost a piece or two in translation so be it. Unless it was an essential ingredient for a potential solution. 

With some types of data, a more robust and fruitful approach is to leave the mess alone. Keep it all. It still needs to be wrangled and organized, but nothing should be thrown out. Instead of an unintelligible mess it becomes a sophisticated well-designed network of data and information on multiple scales. Birger Sevaldson has a name for this kind of data picture: Giga-map 

Applying the the principles of 2D or 3D design (and motion) tames the data and makes it easier for all stakeholders to consume. But it also does something even more important. It allows the designer to expose the data to their visual thinking expertise. In other words, the designerly process of manipulating space and engaging in a kind of structured play and visual experimentation may in fact reveal novel connections and insights that would otherwise have never be discovered. (See giga-map samples here.)