Comparing Market Sentiment – Twitter vs the “Professionals”

If you’ve been following me for more than a few months, you probably know that I’m researching Twitter Sentiment related to the market for my doctoral dissertation.

A few previous posts on the topic:

Last week, I published a quick update via a photo on twitpic to show a few friends and received quite a lot of feedback and response on that chart.

The chart that I shared was a chart of the 21 day moving average of the Bear / Bull Ratio of twitter sentiment..a revised version (current with data up to and including August 28 2012) is below. In the graphs below, the higher the number, the more ‘bearish’ the sentiment…the lower the number, the more ‘bullish’ the sentiment.

The top graph is the 21 day moving average of the Bear / Bull Sentiment Ratio with the average ratio shown as a yellow horizontal line. The bottom graph is the raw Bear/Bull Sentiment Ratio…you can see that it is rather noisy, hence the moving average to smooth it out.

I’ve taken things a step further to look at a longer-term view of twitter sentiment using Weekly Data. For this data, I sum up all data for each week (starting on Monday and ending on Sunday).  The Weekly Bear / Bull Ratio is shown in the top graph using a 5 week moving average of the Weekly Bear/Bull Sentiment Ratio.  The bottom graph contains the Raw Weekly Bear/Bull Sentiment Ratio data.

The question is, is the sentiment useful for understanding future market direction…or is it something that is ‘created’ by market movement.  That’s part of the research that I’m doing now.  I’m working on researching if there is any correlation / causation between twitter sentiment and price movement.  My gut tells me that there is correlation and the data points to this…but is twitter sentiment leading or lagging the market?  That’s the question (at least on of them that I have).

Is the Twitter Sentiment believable/accurate?

One of the things that has bothered me since day 1 of this research is whether the sentiment found via my twitter collection / analysis engine is ‘accurate’.   My analysis is only as accurate as my training data set…and right now, my training dataset shows an accuracy of ~90% using the Python Natural Language Toolkit’s accuracy measures.  So..I feel good there.

But…I wanted to look at comparing my Bear / Bull Sentiment Ratio with other sentiment measures to get a feel for how it might measure up.

I had a moment of brilliance yesterday.   Well…not really brilliance…more like a random thought. But…who’s to say brilliance isn’t really just randomness in the universe 🙂

I decided to compare twitter sentiment to the American Association of Individual Investors (AAII) Sentiment survey data that they release every week.

The latest AAII Sentiment Survey is shown below. I’ve taken the data from the latest survey spreadsheet and created a similar ratio to what I am using (I divide the bearish sentiment by the bullish sentiment).

Taking the AAII data and my Weekly Twitter Sentiment Bear/Bull data, I did a quick comparison between the two.  While the extremes are different, the data  generally follows similar patterns.

Not bad. The levels of the Bear/Bull ratio are different and more pronounced in the AAII survey, but the overall ‘direction’ is similar.

I feel good about this.  AAII reaches out to professional money managers for their sentiment survey…I’m watching Twitter to gather sentiment from people who are talking about the market.  While the data points are exact matches, the directional bias seems to be close.

Look for more on this in the future.

Using Twitter Sentiment for predicting stock price movement

I just finished giving a presentation titled “Will Twitter Make you a better investor?”…and like I always do with these presentations, I recorded one of my rehearsal’s to share.

In this presentation, I provide an overview of my research into using twitter sentiment and message volume as inputs into modeling stock price movements. A quick and dirty linear regression model using Twitter Sentiment, the Number of Tweets per day, the VIX Closing price and the VIX Price change delivers a simple model for the S&P 500 SPY ETF that has an accuracy of 57% over 6 months (tested on out-of sample data). This model was built using data from July 11 2011 to August 11 2011. Note: Accuracy is a measure of predicting the direction of movement.   Being accurate and making money from that accuracy is two different things.

Update:  Please note that the Linear Regression model described in this presentation is far from ideal. When modeling Time Series data, the linear regression model must be used with care due to autocorrelation issues.  

If you don’t want to listen to me yammer, you can jump down to the bottom of this post and take a look at the slides.

The presentation (if you don’t see anything…jump over to Vimeo to watch it there (~30 minutes)):

Twitter Sentiment & Investing – modeling stock price movements with twitter sentiment. from Eric D Brown on Vimeo.

The slides (if you don’t want to listen to me yammer):

Understanding Twitter Sentiment for Investing Decisions

Its been quiet over the last few weeks on here for a reason…

I’ve been finalizing my Dissertation Proposal.

Yesterday (Wed Oct 26th), I successfully defended my proposal. (!!!!!!!!!!!!!!!!!!!!!!!)

I am now officially in the dissertation phase.

While preparing/rehearsing for the defense, I had the foresight to record one my rehearsals and thought I’d share it with you….be forewarned – its roughly 30 minutes long.  If you’d rather just look at my slides, feel free to jump over to SlideShare to review the slides titled “Understanding Twitter Sentiment for Investing Decisions“.

Now…the video.  You’ll have to excuse some of my pauses and “uh” moments – i delivered the final result much better…I promise 🙂

Understanding Twitter Sentiment for Investing Decisions from Eric D Brown on Vimeo.

The Slides (embedded from Slideshare):

Rules and Systems

opinion By the|G|™ on flickrA few months ago, I wrote a post titled How project management made me a better trader/investor where i talk about my experience and skills learned from years as a project manager and how they’ve helped in my trading and investing.

In the world of trading and investing, you learn by failure and you learn by success.  There’s so much to know and comprehend as a trader and investor.  To succeed, you’ve got to be prepared for failure and you’ve got to focus.

In addition…you’ve got to be prepared to follow the ‘rules’.

There aren’t any rules set in stone. To really be successful, you have to make your own rules. You’ve got to find a system or approach that works for you, your investing / trading time frame and your financial situation.

For example – if you want to live off your trading earnings monthly, you may not want to undertake a long term trend following method.  Or – if you only have 15 minutes per day to give to the market, you’d be much better off taking a longer term investing approach and finding / building a system that works for you.

So – these ‘rules’ and your system / approach is important to your success.  Without them, you’ll not really have any serious plan for success as a trader / investor.

The same is true for the world of project managers, no?

Without a formal project management ‘method’, how can you really know where you are going, how you will get there and/or when you are done?  FYI – I don’t care if your using an ‘old’ waterfall method or a ‘new’ agile method for your projects…you have to have a method. You have to have a ‘system’ for accomplishing your project goals.

This system is full of rules. These rules are set in place for a reason…they help direct the project managers, project team and stakeholders to understand what’s happening, why its happening and where the project is going.

Without these rules you’re pretty much hosed. The project will be off track and you can bet you’ll break one of the time, budget or scope areas within the project.  These rules are in place to guide a project manager and team.

Regardless of whether the PM or project team agrees with the rules or the approach, they’re there for a reason. The PM may have an opinion that the plan isn’t correct, but she better not deviate from that plan without first going back to the stakeholders and getting approval to change the plan.

Same goes for the project team…they may have their opinions on how things should be done, but they better not deviate from the agreed upon approach without first circling back to the PM to make sure their new approach doesn’t affect some other team member’s approach or affect the overall project plan.

The same is true in the trading and investing world. As a trader, you have a plan, a system and your rules…and you need to follow them. You may have an opinion about the market, but unless that opinion is worked into your plan with the proper rules for managing risk and entry/exit, you might just get ripped to shreds in the market.

Two of my favorite traders – Ron Roll (on twitter as gtotoy) and Peter Brandt (on twitter as PeterLBrandt) – shared some excellent insight recently about their approach to trading.

Ron shared the following insight on twitter:

[blackbirdpie url=”!/gtotoy/status/103192868109299712″]


On a similar note,  in a post titled When does a victory feels like defeat?, Peter writes:

…an opinion is not a position and a position is not an opinion…This is a tough lesson for novice traders to learn. But it is a lesson that must be learned.

There’s a difference between an opinion and an investment / trade. Without applying your system and rules to that opinion, you shouldn’t make a trade. Same is true for projects…without running your opinion through the approach system, you can’t really tell what risk that opinion might introduce into the project.

In the same article, Peter writes:

Do I periodically override my rules? Yes. Do I usually regret overriding rules. Yes again. My trading rules are based on what I know about market behavior, not on how I feel at any given moment….

Battling the emotional pull to override trading rules is the toughest part of my job. There is not even a close second.

Emphasis mine.

It’d be very easy to change this to focus on projects…a project’s plan and stakeholder agreement (i.e., the rules) are based on years of experience and expertise, not on one person’s opinion. One of the toughest things a project manager (and trader) can do is keep their opinions in check and ensure their systems are followed.

Your system (e.g., project plans, procedures and processes, etc) are there for a reason…start working outside that system and you’ll have hell to pay with project failures, cost over-runs and/or scope creep (or…as a trader…monetary losses out the wazzoo).

Now…unlike the photo I used at the start of this post, your opinion does, in fact, matter. Just make sure you run that opinion through your system and processes to ensure that its in line with reality.

As a project manager or trader/investor – your system and rules are your life…follow ’em or fail.

Image Credit: opinion By the|G|™ on flickr

Stop Chasing. Start Focusing.

chase By rahuldeebee on flickr Most businesses are chasing.

They’re chasing their competitors. Chasing their industries. Chasing their perceptions of their future as well as chasing their past.

Many people within companies are chasing too. Their chasing the approval of their leadership team. Chasing the approval of their managers and of their peers. Some are even chasing recognition from their industry.

But..chasing is dangerous.   Chasing destroys focus,  burns resources and forces a never-ending chase for ‘next ‘.

Chasing and Trading

I’ve mentioned in previous posts that I’ve been working on improving my investing  / trading skills (see Are you ready to accept failure?,  How project management made me a better trader/investor and Revisiting Process – what is the ‘right’ process? for a few examples).

Over the months that I’ve been trading, I’ve learned a lot and  made some money…lost some money too.  Last night I spent some time reviewing my trading journal for the last year. During this review, I noticed  my most profitable trades have been those that have come via my trading system research and from one of my trading mentors.  More importantly, the least profitable trades and/or those that lost money outright were those that I took while chasing something or someone.

My best / most profitable trades have come from a mixture of various trend following trading systems and  methods that I’ve been learning (Check out Michael Covel’s Trend Following books and the Darvas Trading method for examples). With my trend following approaches, I’m able to spend a few minutes a day reviewing trading setups and managing them if needed. These methods have clear entry and exit rules…so it makes it very easy to determine what to trade, when to enter the trade and when to exit the trade.

Additionally, I’ve had great success following the advice of Ron Roll (aka Goat – @gtotoy on twitter) who runs The one thing Ron always talks about is not ‘chasing’ a trade….don’t throw your money into a stock just because everyone else does. Ron takes his time and teaches everyone else to do the same…focus on those stocks that are setup for your trading style / approach and when its right, pull the trigger on the trade.   There are others out on the web and twitter that I’ve learned from over the last year but Ron has been the biggest help to me in understanding how to “Plan your trade, trade your plan”.  So..a special “thank you” to Ron for the lessons I’ve learned.

As I mentioned, my worst trades have come while chasing. I’ll see someone post something on twitter about a stock ‘moving’ or being ‘on fire’ and think ‘yes…I need to catch that!’.  The majority of the time, this approach ends up with me losing money – because i chased someone else into a trade.   In addition, the time that I spend reading tweets and trying to ‘chase’ these recommendations is time away from focusing on what my next step should be in my trading plan.

For me and my trading style and abilities, chasing kills my profitability and my focus.  Looking through my trading journal, I couldn’t find a single ‘chasing’ trade that made me money.

Chasing is Dangerous

Looking over my trading journal it was very apparent that my best trades were those that were planned out. They were  thought through with entry and exit criteria and I know exactly what i’m getting into. I managed my risk and set everything up to ensure that I gave the trade the best chance for success.

This happens in business too…we plan out projects, manage the risk within those projects and we try to give that project the best chance for success.

Or do we?   We should…but a lot of times, we don’t.

We spend our days chasing status reports.  Chasing resources and money.  We spend our days chasing projects and people. We chase strategies and tactics that are supposed to help us improve.

Stop chasing.

Stop chasing your competitors and your industry. Stop chasing your friends and colleagues. Stop chasing the latest / greatest technology, methods and best practices.

Find out what you and/or your business needs to succeed and make it happen.  Focus on your strategy, your people and your approach. Focus on what it will take to make yourself and/or your business better.

Stop chasing. Start focusing.

Image Credit: chase By rahuldeebee on flickr

How project management made me a better trader/investor

dollar sign $  By Leo Reynolds on flickr

dollar sign $ By Leo Reynolds on flickr

I’ve recently moved back into the world of trading stocks and options.  Most of what I’m doing is small stuff but I’m enjoying it and making some money.

Some quick history – me and trading/investing

In the past, I’d been into investing quite heavily with the ‘buy and hold’ approach.  Find a company with good fundamentals and buy it at a good price.  Then hold it and wait for it to go up.

Not a bad long-term approach. I still have some core holdings that are long-term but I never just buy to hold any longer…I buy with a plan in mind. I know when I’ll sell. I know how much I’m willing to loose and how much I’m willing to earn from the trade.

Today, most of my investments are short trades that i hold for less than a week. Many are held overnight for quick in-and-out trades (call swing trades in the trading world). I do some daytrading (buy/sell in same day) but not a lot since I don’t have enough capital in one account to get past the daytrading rules.

Project Management & Trading/Investing?

So…how has project management helped me to be a better trader?

Simple…as a project manager, you focus on three things: Budget, Timeline and Scope

To be successful managing projects, you’ve got to plan well, spend well and execute well. You’ve got to know what you want to do, how much money you have to do it and then execute properly to accomplish the goal(s).

The same is true for trading.

To be a successful trader, you’ve got to plan well, spend well and execute well.  Just like managing projects, you’ve got to know what you want to do, how much money you have to do it and then execute properly to accomplish the goal(s).

In addition, just like being a good PM, you’ve got to know how to manage risk.  Same is true for traders….risk management is key to a long trading career.

There’s one major difference between trading and project management though…..and its a big one:

Trading uses your money….project management uses your bosses money.

It’s usually easier to spend someone else’s money:)

If you mess up on a project, someone can always allocate more money to the project to cover you…you mess up a trade, there’s nobody there to help….that’s where your budget and risk management come into play.  Put too much into a bad trade and you blow up your account.

Trading is all about having a plan and then working that plan.  Find the stock setups that work and then look for the right entry/exit.  Think about what you want to do and be patient and let the plan work.  Sounds similar to what a project manager does, no?

If anyone out there’s interested in learning more about trading, feel free to drop me a line.  For much more knowledge folks on the subject, check out or some of these good traders, investors and thinkers in the space:

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.