Big Data Doesn’t Require Big Systems

big-dataI’ve written many times about big data and the benefits and challenges that come along with attempting to ‘do’ big data.

The one thing I haven’t said much of is how easy it is to ‘do’ big data. It really is fairly straightforward and inexpensive to get started in the big data realm.

Before I go any further, let me make a clarification – if you are going to use large amounts of data across multiple locations and you’ll be doing large-scale analysis, you’ll most likely need top spend some money on platforms, tools and people. That said, you don’t need to go all-out if you are new to big data and are just trying to get a grasp on what it is, how to do it and how data analysis might help your business.

In order to start ‘doing’ big data at small scale, you really don’t need a lot of expensive and fancy systems and tools…you only need to have a few capabilities to collect, store and analyze data. This can be as simple as taking data you’ve already collected and using Excel to analyze and visualize or it can be as complex as your skills and technological abilities allow.

I know a midsized business here in the Dallas area that uses Excel along with a connection to their SQL Server to do all of their data analysis. They are currently running in the Terabytes of data and have built their data analysis process to be very lean. Regardless of this ‘lean’ approach to big data, they have been able to ask and answer some very interesting questions related to their business over the last year, which in turn has brought new revenue streams and new highs in annual revenue and profit margin.

In addition to the midsized business just described, I recently helped a startup with their big data processes and systems. While designing their business, they budgeted about $500K in the first five years for their big data initiatives. While I can’t describe what exactly they are doing, I can tell you their entire business is built on analyzing social streams for ‘knowledge’. To do this, they require a heavy duty analytical engine with near-real-time capabilities, hence the large costs. After some analysis of their approach, they were able to start with an estimated $20K in expenses for the first year and then grow their platform and storage requirements by about $20K per year. Compare the original $500K budget over five years to $100K over the same time period and you can see some significant savings.

These stories are used as examples to describe two significantly different approaches to doing big data. One approach uses pre-existing technologies and capabilities and has led to millions of dollars of newly identified markets while the other requires some fairly large platforms and systems (but not as large as planned for).

Don’t let big data scare you. You don’t need to ‘go big’ to do big data. You can start small and build your way into larger systems and platforms as your needs progress.

IBMThis post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don’t necessarily represent IBM’s positions, strategies or opinions.

hit counter

2 responses to “Big Data Doesn’t Require Big Systems”

  1. James Lawther Avatar


    I agree

    My personal perspective is that very many organisations already have plenty of data at their finger tips that they could (but don’t) use.

    Far better to learn how to walk before you jump on a motorbike.


    1. Eric D. Brown Avatar

      Thanks James….great perspective.