Big Data, Icebergs and Janitor Work

Welcome to the sexy world of big data. A world in which data scientists make bold predictions about the future of business. They make millions of dollars and spend their days jet-setting around the world answering the call of ‘big data’.

So…maybe it really isn’t all that sexy. Data scientists might make decent money but most aren’t flying around the world “jet-setting”. Most data scientists are sitting in a room somewhere staring at their computer screens and the many gigabytes (or terabytes) of data.

They New York Times called this work “janitor work” and claim that:

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

Based on my experiences, I’d say those numbers are accurate. Most of my time spent in the world of big data is spent collecting and working with data to get it into a form that can be easily analyzed using automated scripts and visualization systems.  Doing the ‘janitor work” is important. It helps you to get a feel for the data and helps you spend time with the data to better understand the ‘context’ of the data.

Most of the work undertaken in the world of big data is done below the surface. Most people don’t see the many hours of work put into cleaning and organizing data. They only see the output.  Big data is like an iceberg…the outcome of big data analysis is really a small part of the overall effort put into the analysis of data. You only ‘see’ a small portion of the work that goes into big data and you really only see a small portion of the data itself.

Because big data is like an iceberg, the work most data scientists do will never really be appreciated by anyone outside the data science world. Because of this, the work you do in the data world will be under-appreciated and I believe, in the long run, under-rewarded.  Sure, data scientists are making decent money compared to other career choices but when you stop and think about the value that a good data scientist can bring to an organization compared to the money paid to that data scientist, I think the organization is coming out ahead.

Moving into a job in the world of big data isn’t really that glamorous. Sure, you may be part of the new ‘buzzworthy’ world of big data but the work you do day-to-day will never be glamorous.  Make sure you’re OK with doing ‘janitor work’.  Make sure you’re OK spending most of your time working on the unseen pieces of the iceberg.