Losing Big with Big Data

big_data1I help companies use big data to work better.  I love what I do and I love seeing companies and people succeed with big data.

That said, I’ve seen my fair share of companies (and people) lose with big data too. Most of these ‘losses’ aren’t due to bad data or poor usage of analytical tools. Most of these losses can be traced to a few simple decisions that were made when a big data initiative was being planned.

There are plenty of things that can go wrong during a big data ‘project’ but most of the real problems that can cause large problems have to do with the ‘strategic’ side of big data projects.  Tactical problems can be overcome but it is much harder to overcome poor planning and strategy.

From a strategic standpoint, there are quite a few questions that need to be answered when planning for big data. A few of the big ones are:

  • Do you have the right people?  Big data requires different people than other data analysis projects.   Big data isn’t data warehousing. With big data organizations should look for ways to get the data and the analysis of that data into the hands of the people closest to the problems trying to be solved.  With that being the case,  you need to have people throughout the organization that are curious, interested in analyzing data and willing to learn. Additionally, you’ll need IT professionals with the same skills and interests.
  • Is your project too ‘big’? Big data can bring big changes to an organization but if you bite off more than you can chew, you may only be wasting time and money in your big data initiatives. There’s nothing wrong with starting small with big data….better to start small, learn and possibly fail then to jump into big data with a great deal of money and time invested and fail. Find projects that let you get some big data experience under your belt without spending a great deal of money. Once you’ve got some experience (and some wins), then start working on larger and larger projects
  • Are you willing to invest for the long term?  Big data isn’t something you put money in one time and hope to be successful.  You can’t just ‘pay once’ and be done. With big data, you’ll need to continue to pay for new systems, new technologies, new skills and new people. 
  • Are you willing and able to open up your data? Some of the most successful companies using big data that I’ve worked with (and heard about) are ones that have opened up their data to their organization. This doesn’t mean that you should allow everyone unfettered access to all your data but it does mean finding ways to allow access to data with proper access rights and security.  Using proper data management and data governance systems and methods allows you to open up access to your data to anyone that needs access.   With open data access you’ll get more eyes on your data and more insight into ways to solve your problems.

I could probably continue with many more questions/issues that need to be addressed but these four are key to getting started on the right path in big data.

I’ve worked with a few companies who didn’t answer these questions before starting up big data initiatives. In some cases the failure was small and easily managed but in others the failure was quite large and expensive.  These organizations lost money, time and revenue from the failure of these projects.

Even more importantly (and perhaps more dangerously), they lost quite a bit of confidence in their ability to ‘do’ big data. They became very very concerned about planning for any future big data initiatives because they felt that it was ‘just too hard’.    But…it really isn’t all that hard.

You don’t have to lose big with big data.  Big data is complex and difficult, but with the proper planning and strategic thinking, you can prepare for many of the challenges that you’ll face in your big data initiatives.

Links for July 7 2011

Are you ready to accept failure?

Success 02 By PaDumBumPsh on flickr

Are you ready to accept failure?  Can you live with yourself if you don’t succeed?

When I was growing up, winning was a big thing.  You won or lost. You won at sports.  You succeeded at school.  Or you didn’t.  There wasn’t anything good about losing…no ‘participation’ trophies…no recognition for losing.

Succeeding and winning was everything.

Everyone fails though.   Nobody is perfect.  Nobody succeeds at every thing they do. With success comes failure. When one person succeeds, someone else is most likely to fail.  That’s not a bad thing though…failure is acceptable if you learn from that experience, right?

Take an example from my new found interest….the world of trading & investing. In the investment world, nobody succeeds all the time.  Heck…most traders are happy if they have 50% win ratio (I’m currently at a 53% win ratio…and I’m ecstatic about that) The key to succeeding long term when you ‘only’ win half your attempts is to ensure you manage your risk appropriately. Peter Lynch, that icon of the investing world, sums up the idea of success / failure in the investing world with:

In this business if you’re good, you’re right six times out of ten. You’re never going to be right nine times out of ten. (HT @Fibline for the quote)

Failure is perfectly acceptable, if you’ve approached that failure correctly and learn from your failure.

Are you allowing yourself to be right six times out of ten?  Are you happy with a 50% win ratio?

How about a having win rate of 5% (1 win out of 20 attempts)?  Would that you make you happy…or angry? Most people would absolutely hate a 5% win rate.

But…some folks can accept that rate and move on.  They learn from their successes and failures.

Take OfficeMax as an example. Scott Brinker recently shared a story about OfficeMax’s digital marketing efforts in a post titled Experimental Marketing: 1 out of 20 ain’t bad.  OfficeMax had 1 successful website (their “Elf Yourself” website) out of 20 attempts.  They built 20 different websites, each with a reason behind it and expectation that it would work.

But…out of those 20, only 1 of them – the Elf Yourself site – succeeded.  The remaining 19 were quietly turned off.

Would a 5% win rate be OK with you? Would it be OK with your company?

Would you or your company ever allow a project to kick off that had a 1 in 20 chance of succeeding?  Probably not…but what if you kicked off 20 projects, each with compelling business cases, and only 1 succeeded?  Would you take the time to understand what worked and what didn’t in those 20 projects or would you fire the people in charge of those projects?

Success is wonderful.  Failure is just as wonderful…if you learn from it.

Now…in the trading world, you wouldn’t really want a 5% win rate…you’d quickly run out of money.  In the world of Marketing, IT and Technology…a 5% win rate isn’t ideal either..but sometimes you’ve got to have the guts to make a decision that might just lead to a low success rate on projects…and you might just learn a great deal about how to get better the next time.

Today’s IT project success rate is hovering around 30 to 35 % depending on what study you read.  Are organizations & IT groups learning from the failures of those projects that aren’t succeeding?  In my experience, no….those failed projects are written off and never looked at again.

So…are you prepared for failure?  Ready to accept it and learn from it?  If you want to succeed more in the future…you should be.

Image Credit: “Success 02 by By PaDumBumPsh” found on flickr. Make sure you jump over and read the small print…It’ll make you giggle…at least it made me 🙂

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.