Losing Big with Big Data

I help companies use big data to work better.  I love what I do and I love seeing companies and people succeed with big data.

That said, I’ve seen my fair share of companies (and people) lose with big data too. Most of these ‘losses’ aren’t due to bad data or poor usage of analytical tools. Most of these losses can be traced to a few simple decisions that were made when a big data initiative was being planned.

There are plenty of things that can go wrong during a big data ‘project’ but most of the real problems that can cause large problems have to do with the ‘strategic’ side of big data projects.  Tactical problems can be overcome but it is much harder to overcome poor planning and strategy.

From a strategic standpoint, there are quite a few questions that need to be answered when planning for big data. A few of the big ones are:

• Do you have the right people?  Big data requires different people than other data analysis projects.   Big data isn’t data warehousing. With big data organizations should look for ways to get the data and the analysis of that data into the hands of the people closest to the problems trying to be solved.  With that being the case,  you need to have people throughout the organization that are curious, interested in analyzing data and willing to learn. Additionally, you’ll need IT professionals with the same skills and interests.
• Is your project too ‘big’? Big data can bring big changes to an organization but if you bite off more than you can chew, you may only be wasting time and money in your big data initiatives. There’s nothing wrong with starting small with big data….better to start small, learn and possibly fail then to jump into big data with a great deal of money and time invested and fail. Find projects that let you get some big data experience under your belt without spending a great deal of money. Once you’ve got some experience (and some wins), then start working on larger and larger projects
• Are you willing to invest for the long term?  Big data isn’t something you put money in one time and hope to be successful.  You can’t just ‘pay once’ and be done. With big data, you’ll need to continue to pay for new systems, new technologies, new skills and new people. 
  
• Are you willing and able to open up your data? Some of the most successful companies using big data that I’ve worked with (and heard about) are ones that have opened up their data to their organization. This doesn’t mean that you should allow everyone unfettered access to all your data but it does mean finding ways to allow access to data with proper access rights and security.  Using proper data management and data governance systems and methods allows you to open up access to your data to anyone that needs access.   With open data access you’ll get more eyes on your data and more insight into ways to solve your problems.

I could probably continue with many more questions/issues that need to be addressed but these four are key to getting started on the right path in big data.

I’ve worked with a few companies who didn’t answer these questions before starting up big data initiatives. In some cases the failure was small and easily managed but in others the failure was quite large and expensive.  These organizations lost money, time and revenue from the failure of these projects.

Even more importantly (and perhaps more dangerously), they lost quite a bit of confidence in their ability to ‘do’ big data. They became very very concerned about planning for any future big data initiatives because they felt that it was ‘just too hard’.    But…it really isn’t all that hard.

You don’t have to lose big with big data.  Big data is complex and difficult, but with the proper planning and strategic thinking, you can prepare for many of the challenges that you’ll face in your big data initiatives.