“Big data” is all the rage these days. Done right, it promises the world – infinitely optimized decisions based on “real” data. The problem is that humans aren’t data. And while they certainly fall into patterns, they can also be oddly chaotic at times – especially when stuff hits the fan. A canonical example is the US Military during WW2. The Army had published doctrine, and the Germans studied it. They knew what we were supposed to do. So why did the US win? Turns out that Soldiers often didn’t follow doctrine, instead improvising based on the situation. So knowing the answer ahead of time didn’t convert to an operational advantage.

I view big data as a related opportunity for misunderestimating what technology can do. I’m sure there are data scientists who would argue that there is no such thing as an unexpected behavior – that those fall into patterns as well, and if we only had more data and better algorithms we’d be able to pick them up. I’m holding out for the idea that humans and their perfectly imperfect biochemistry will always result in oddities and outliers that defy big data. And in a era where big data will rule decision making, perhaps these human anomalies will be the real winners?

Perhaps there is hope for artists and musicians being highly valued in society after all? Time will tell.