To have a great analytics culture, you need a great communications culture

Employee-communicationWhen you read about big data and/or data analytics projects and systems, it is rare that you also read bout communicating the outcome of those projects. Without the ability to communicate the results of any analysis to the broader business, most big data / analytics projects are doomed to mediocrity…or even failure.

The quantitative mind is a great one. It is one that I’m very familiar with and one that I wholeheartedly support.  The ability to take a data set, analyze that data and create new information and knowledge from that data is an extremely important skill for people and organizations to have.

Just as important is the skill to be able to convert the outcome of any quantitative analysis into something that is easily digestible by people throughout an organization.

Take, for example, the world of academia.  There are many really smart people performing research within universities and research facilities. These people conduct research and then publish the outcomes of that research in academic journals to share their new-found knowledge with others.

Have you ever picked up an academic journal/article? These articles are generally well-written and delivered in formal academic styles but they aren’t exactly ‘easy reading’.   They are meant to be used for academic reporting within academic circles. They are also used within industry but most practitioners that read these journals and articles are usually people with similar education and experience as those folks who are writing / publishing these articles.

What happens when a finance manager picks up the Journal of Finance paper titled “Determinants of Corporate Borrowing?” Will they easily understand what the paper is trying to communicate?  Let’s take a look at a portion of the abstract of the paper:

Many corporate assets, particularly growth opportunities, can be viewed as call options. The value of such ‘real options’ depends on discretionary future investment by the firm. Issuing risky debt reduces the present market value of a firm holding real options by inducing a suboptimal investment strategy or by forcing the firm and its creditors to bear the costs of avoiding the suboptimal strategy. The paper predicts that corporate borrowing is inversely related to the proportion of market value accounted for by real options. It also rationalizes other aspects of corporate borrowing behavior, for example the practice of matching maturities of assets and debt liabilities.

I would argue that anyone – given enough time – could understand what that paragraph is trying to communicate, but in the fast-paced world of business, does anyone really have time to sit down and study this paper?  I doubt it.  Most will call up a consultant and ask to help better understand the optimal approach to corporate debt.  What is that consultant going to do?  She will take her experience as a consultant (and in finance/banking), study the business, literature and best practices and then make a recommendation to the business on what they should do. If the consultant is any good, these recommendations will be provided in an easy to understand document that can be implemented effectively within the organization.

The same approach needs to be taken with data analytics.  We can’t just throw a spreadsheet or chart over the wall at the business and expect them to understand what the data is telling them or what they should with that data. I see a lot of this these days though. A company will implement a new big data project, perform some analysis of the data and then provide the output of the analysis in pretty charts and tables but very rarely are there deep, meaningful discussions and analysis about what that data is really telling the business and/or what the business should do based on the data analysis.

Now, you may say that good data scientists / analysts already do this…and you’d be right. But, not everyone is a great analyst nor is it a skill set that most organization’s are hiring for these days. When I talk to clients about big data, they talk about the need to get the best hardware, software and analytical skills…but they rarely talk about the need to find great communicators.

Companies regularly spend millions of dollars on the ‘hard’ costs for big data and data analytics. They’ve even begun spending a good deal of money on the ‘soft’ costs to get their people the best training available so they can be the best data analysts available but it is rare that they spend much money on communications training.

The funny thing about this particular topic is that most data scientists consider themselves to be good communicators.   In my experience, the really good ones are…but the majority of the ‘new’ data scientists struggle with this aspect of their job.

If you want to be a great data scientist, become a great communicator and storyteller. As a data scientist, if you can’t communicate in a way that is informative and useful to the business, the work you do in the ‘quant’ world isn’t that valuable to the company.  The same can be said to the business in general – if you want a great data analytics culture, build a great communications culture. You can’t have one without the other.

My Doctoral Dissertation Final Defense – Almost done

I am now one step closer to finishing my doctorate. On Friday Oct 31, I defended my dissertation. The video of the presentation during the defense is provided below. I now only have to get a few documents signed and format my dissertation for publishing and I’ll be completely finished.

The title of my dissertation is: “Analysis of Twitter Messages for Sentiment and Insight for use in Stock Market Decision Making.”

The video is a bit over 1 hour and 12 minutes long. I cut out the question and answer session for the sake of brevity.

Using Twitter Sentiment in the Stock Market from Eric D Brown on Vimeo.

This video is a copy of my Doctoral Dissertation defense. The topic: Analysis of Twitter Messages for Sentiment and Insight for use in Stock Market Decision Making.

Dissertation Title: “Analysis of Twitter Messages for Sentiment and Insight for use in Stock Market Decision Making”


Finding Value in Data

value-big-data-150x150If you ask 100 people to define the value that data brings an organization, you’ll most likely get 150 different answers. Yes…that’s right…150 answers. You’ll hear a few good succinct responses but most will give you a few different answers with examples of how their organization ‘uses’ data.

There’s a problem with that response though. Using something doesn’t mean it has value. Doing data analysis isn’t delivering value; its just doing data analysis.

The value of data comes from the purposeful analysis of that data. Additionally, you can’t analyse data and hope to find value without an idea in mind of where that data comes from (i.e., context) and how you hope to be able to use that data (i.e., strategy).

Approaching data analysis without context and strategic purpose is similar to getting on the highway with a full tank of gas and driving without a destination in mind. Sure, you’ll get somewhere but you won’t really know if that ‘somewhere’ is the ‘right’ place.

A caveat to the contextual and strategic aspect of data analysis is that if you are ‘doing’ big data the right way, you should find more questions than answers. Your strategic thinking and data context should continue to drive the analysis and use of data within the organization but you should be willing to take whatever questions and answers you find.

Some people would argue that part of data science and big data is finding new questions and insights that you didn’t know you were looking for. In fact, you never really know what you’ll find when you start digging into data; but there’s a difference in digging into data without some sort of roadmap and purposefully analysing data.   If you don’t have some form of strategy behind your analysis, you won’t really know when you’ve found something of importance.

There is value in data but the real, long-term value of data comes not from just the data itself but from the purposeful analysis of that data.

A look at Twitter messages in 2012 mentioning $SPY and S&P500 Symbols

twitter-bird-blue-on-whiteCross Posted at

While working up my data analysis chapter of my dissertation, I came across some interesting tidbits of information and thought I’d share.

Nothing here is earth-shattering and there’s not much I (or you) can do with this…but I thought it interesting and hope someone else out there does too. I’ve shared other findings before – and continue to share my daily Bear/Bull Ratio via my Trade The Sentiment site, which is an outcome of this research.

For the data collection phase of my dissertation, I collected Twitter messages for all stocks in the S&P500 index and the SPY ETF itself.  There are many great pieces of knowledge that I’ve gathered from this work – some I’ve shared but most I won’t share because I need something to put into the dissertation. 🙂

So…here’s some data that you might find interesting (or maybe you won’t). Without further ado – and without interpretation, here you go:

SPY and all symbols in S&P 500 Index

Dates: Jan 1 – Dec 30 2012

  • Number of Twitter Messages Captured: 1,655,962
  • Number of Symbols: 501 (S&P 500 + SPY)
  • Number of days messages captured: 361
  • Number of Twitter users: 224,499
  • Average Messages per day: 4,587.15
  • Average Messages per user: 7.38
  • Date with Highest message volume: December 5 2012
  • Symbol with most Mentions: AAPL (620,964 messages or 37.5% of messages)
  • Symbol with most Bearish Mentions: AAPL with 98,402 messages with bearish sentiment
  • Symbol with most Bullish Mentions: AAPL with 78,353 messages with bullish sentiment
  • User with most Tweets: SeekingAlpha
  • Top 10 users account for 128,703 messages or 7.77% of messages
  • Top 25 users account for 197,878 messages or 11.95% of messages
  • Top 50 users account for 278,846 messages or 16.84% of messages
  • 50% of messages were sent by 849 Twitter users or 0.38% of users
  • 80% of messages were sent by 14,049 Twitter users or 12.27% of users

 Top 50 captured Twitter Users:

  1. SeekingAlpha
  2. BigTicks
  3. thefinancepress
  4. wallstCS
  5. CPUStocks
  6. gasoilstocks
  7. ADVFNplc
  8. MarketCurrents
  9. StockRecaps
  10. PerforM84697233
  11. TheStreet
  12. simplestockqtes
  13. Tradified
  14. SAI
  15. BigChipStocks
  16. pennystockguys
  17. DJThistle
  18. RetailerStocks
  19. TradingGuru
  20. boogidown
  21. USwwwStocks
  22. lluccipha
  23. MNYCx
  24. investorpoint
  25. takingstock614
  26. tradingview
  27. stockticks
  28. 1nvestor
  29. ForTraders
  30. FastFoodStocks
  31. StockTwits
  32. some_win
  33. ValueStocksNow
  34. PiggyStocks
  35. Insider_Trades
  36. 61point8
  37. BlueFielder
  38. tlmontana
  39. stockguy22
  40. Phil_Goodship
  41. LaMonicaBuzz
  42. Jamtrades
  43. businessinsider
  44. BUDDIEE18
  45. ZolmaxNews
  46. OneChicago
  47. olyant75
  48. onebrow1
  49. DeidreZune
  50. bored2tears

Cross Posted at