Eric D. Brown, D.Sc.

Data Science | Entrepreneurship | ..and sometimes Photography

Tag: sentiment

Using Twitter Sentiment for Intraday Signals

This is a cross-post from Trade The Sentiment. Originally published as Using Twitter Sentiment for Intraday Signals.

While most of my research on Twitter Sentiment has been for use on larger time-frames (Daily, Weekly, etc), I’ve been very curious about using sentiment for intraday signals.

I finally found some time to hack together a script that would look at sentiment data intraday…and now i’m a bit unhappy that I did…because I’m fascinated with the intraday signals I’m seeing.

An Example

On Feb 25, we saw a nice little sell-off. The S&P500 gapped up to open the day and sold off for the remainder of the day for a loss of almost 38 points on the $SPX.

The following day, Feb 26, saw another move down until about mid-day when the markets ‘turned’ and started heading up…and we saw a retrace of about 40 points over the next three days.

Everyone seemed to be looking for a breakdown on Feb 25 and Feb 26 but it didn’t happen.

As luck would have it, over the weekend I had built my script to look at intraday sentiment.  It was a quick hack (like most of my stuff) that allowed me to run a quick query to see what sentiment looks like at time “now”.     On Feb 25, I was occasionally calling out the the intraday sentiment values in @gtotoy’s trading room over at DayTraderBootCamp (you should join if you aren’t a member…some GREAT traders there).

As the day wore on, I was noticing the sentiment was getting much more bearish…to the extent that it was in ‘bearish extreme’ levels by a large margin. At one point, the sentiment for the day was around 1.5 or so (1.0 is neutral, anything over 1.25 is considered a bearish extreme).

Twitter users were extremely bearish on the market on Feb 25 and the morning of Feb 26.  On Feb 26th at 9:30 Central and again at 11:30 Central, there were 2 large bearish sentiment spikes…those times marked the same times that we saw the bottom in the market on that day (looking at at 15 minute chart).

At the time, I didn’t have the sentiment loaded up into a platform to view it against price action…but I’ve since fixed that.

Take a look at the following….a chart showing a 15 minute candle chart of SPY Action (in green), Intraday Sentiment sampled in 15 minute increments (in Orange) and the 21 period EMA of the Intraday Sentiment (in Magenta).

You’ll notice that the sentiment chart is fairly noisy…but the 21 EMA is much cleaner and provides a couple nice ‘signals’ over the last few weeks. On the 21 EMA chart (bottom pane), you’ll see a Red oval that highlights a Sell signal with Bullish Extreme on Feb 19  and a Green oval that highlights a Buy Signal on Feb 25 / 26th with Bearish Extreme readings).

Note:  a few days of data don’t make something useful.  It could be pure happenstance that the below signals did what they did…but I’ll  be looking at more historical data to see what I can find.

Click image for larger version

k5vAoWU

Comparing Twitter Sentiment vs Random Numbers for Entry Signals

This is a cross-post from Trade The Sentiment

One of the concerns that I have of my research into using Twitter Sentiment for entry signals in the market has been that the market has, in general, been in an up-trend over the last year.   With that up-trend, there’s a good change that any entry, made enough times over the course of the year, would have resulted in a profitable trading year.

So…I developed a basic random number generation script in Python that generates a random signal between 0 and 2 to try to mimic the ranges found in the overall market bear/bull sentiment.

With this random signal, I then used the same strategy for an entry signal used in my Twitter Sentiment entries.   The comparisons are below

Trading Rules

  • Enter when Bearish Extreme found in Raw Bear/Bull when Sentiment (or Random Signal) > 1.1
  • Buy 500 Shares of the SPY ETF. This strategy is Long Only.
  • Hold for 6 trading Days.
  • Timeframe: Nov 1 2011 to Nov 30 2012
  • Commission = $10
  • Slippage = $0.05 per share

Using Tradestation, I ran my strategy with both the Twitter Sentiment Signal and the Random Signal without optimizing any of the strategy inputs.  Both systems closed out their last trade before Friday so no trades are held.

The outcome of both systems are provided below.  Below the table, I’ve provided the

Note: The Buy and Hold Rate of Returns are different due to the date of the first trade made by each system. Trade #1 for Random Signals was taken on Nov 9 2011 while Trade #1 for the Twitter Sentiment was taken on Nov 14 2011.

Based on this particular test, I’d say the twitter sentiment entry signals are much more robust and accurate. This signal delivers more profit, more return and much less drawdown than the random signals entries.

Each strategy’s entry / exits are shown in the graphs below.

Twitter Sentiment Signals

Random Signals Entries

This is a cross-post from Trade The Sentiment

Can Twitter Sentiment be used to generate buy / sell signals?

This is a cross-post from TradeTheSentiment.com

Before we get started…its VERY easy to look back and say “yes…that would have been a buy signal”.  If the market worked with hindsight, we’d all be millionaires.

For this approach, I’m going to use the thought that the masses are ‘wrong’.  That is…when Sentiment gets extremely bearish or bullish, I’m going to go the opposite direction.  If Extremely Bearish, I’m going Long. If Extremely Bullish, I’m going to short.

Finding the “extreme” is going to be a very subjective approach. For the purposes of this study, I will use the Daily Bear / Bull Sentiment chart. When the sentiment spikes over a  2.0 (a level of 2.0 on the Bear / Bull chart means the sentiment is twice as bearish as bullish), I will go long.

For this study, I apply this strategy to the S&P 500 ETF – SPY.

The entry criteria:

  • An “extreme” is found in the Bear / Bull Daily Sentiment Chart. This is defined as any reading over 2.0.
  • Long only. When an an extreme is found, I’m going to go Long the underlying stock at Market Open the next day. All orders will be entered as “market at open” orders.
  • # of Shares = 500.
  • Commissions = Not included.
  • Slippage=10 cents (included on entry and exit)
  • Stop = $2 (trailing) (e.g., if entry is at $140.00, the stop is at $138.00 and it moves up as price moves up).
  • If I am still in the trade upon a new extreme, no action is taken.
  • Timeframe: Nov 1 2011 to Sept 18 2012

SPY

Peaks are found on the following days (review  signals above the red horizontal line in the bottom pane of the chart)

  • Jan 9 2012
  • March 19 2012
  • April 16 2012
  • June 4 2012
  • August 12 2012
  • August 24 2012
  • August 26 2012
  • August 27 2012
  • Sept 4 2012

With the above dates of extremes in mind, let’s take a look at how an investment strategy using these peaks would do.

Trade #1

Extreme on Jan 9. On Jan 10, go long 500 shares at Market Open.  SPY opened at 129.39. With slippage,  Order is filled at 129.49.  Stop set at 127.49.

Trade lasts 13 days and closes on Jan 27 when Trailing stop is hit for a Gain of +$1.56 per share or 1.20% return on investment.

Trade #2

Extreme on March 19. On March 20 2012, go long 500 shares at Market Open. SPY opened at $140.05. With slippage, Order is filled at $140.15. Stop set at 138.15.

Trade lasts 7 days and closes on March 28 when Trailing stop is hit for a Loss of $0.51 per share or a total return of -0.39% return on investment.

Trade #3

Extreme on April 16. On April 17 2012, go long 500 shares at Market Open. SPY opened at $137.94. With slippage, Order is filled at $137.94. Stop set at 135.94.

Trade lasts 5 days and closes on April 23 when Trailing stop is hit for a Loss of $1.41 per share or a total return of-1.09% return on investment.

Trade #4

Extreme on June 4. On June 5 2012, go long 500 shares at Market Open. SPY opened at $127.85. With slippage, Order is filled at $127.95. Stop set at 125.95.

Trade lasts 4 days and closes on June 8 when Trailing stop is hit for a Gain of $3.42 per share or a total return of 2.64% return on investment.

Trade #5

Extreme on August 12. On August 13 2012, go long 500  shares at Market Open. SPY opened at $140.70. With slippage, Order is filled at $140.80. Stop set at 138.80.

Trade hasn’t closed yet as stop hasn’t been hit.  As of Sept 18, SPY closing price is $146.49. Stop is at $144.94. Current Gain of $4.14 per share or 3.2% return on investment.

No additional Trades taken since Trade #5 is still active.

Outcome of Strategy

5 trades, 5.56% return over 8 months.

In the same timeframe, the SPY ETF, if you were to buy 500 shares of SPY on Jan 10 (the date of the first trade) at the same price of my purchase @ 129.49 and hold them without a stop you’d have a gain of $17.13 per share for a 13.23% return.  That said…you would have also seen a pullback to below your entry point (low of 126.48 on June 4 2012). Would you have stayed in that trade if it had gone against you $3?

Now…I’m not a “buy-and-hold” person. Too much stress. If I can make 5 trades with an average hold time of 11 days and pull out 5.56%, I’d be pretty happy.

That said…if I changed my trailing stop from $2 to $5…the outcome would have been different.   With a trailing stop of $5, this approach would have made 3 trades for a total gain of $25.15 or 19.42% return and an average of 39 days per trade.  Not bad, eh?

Now…this is by no means “proof” that sentiment can be used as a buy signal.  During this uptrending market, I could probably have picked any day at random and run this same analysis and got similar results. But…you can say that about any strategy, no?

Note: For those who point out that commissions aren’t included in this calculation, please note that a $50 charge for commissions ($10 per trade round-trip) will not significantly alter the outcome of this strategy.

In the next few days/weeks I plan to share more of these ‘applications’…stay tuned.

If you want to keep up with more of my market related topics, jump over to my Market focused blog at TradeTheSentiment.com.

This is a cross-post from TradeTheSentiment.com

Comparing Market Sentiment – Twitter vs the “Professionals”

If you’ve been following me for more than a few months, you probably know that I’m researching Twitter Sentiment related to the market for my doctoral dissertation.

A few previous posts on the topic:

Last week, I published a quick update via a photo on twitpic to show a few friends and received quite a lot of feedback and response on that chart.

The chart that I shared was a chart of the 21 day moving average of the Bear / Bull Ratio of twitter sentiment..a revised version (current with data up to and including August 28 2012) is below. In the graphs below, the higher the number, the more ‘bearish’ the sentiment…the lower the number, the more ‘bullish’ the sentiment.

The top graph is the 21 day moving average of the Bear / Bull Sentiment Ratio with the average ratio shown as a yellow horizontal line. The bottom graph is the raw Bear/Bull Sentiment Ratio…you can see that it is rather noisy, hence the moving average to smooth it out.

I’ve taken things a step further to look at a longer-term view of twitter sentiment using Weekly Data. For this data, I sum up all data for each week (starting on Monday and ending on Sunday).  The Weekly Bear / Bull Ratio is shown in the top graph using a 5 week moving average of the Weekly Bear/Bull Sentiment Ratio.  The bottom graph contains the Raw Weekly Bear/Bull Sentiment Ratio data.

The question is, is the sentiment useful for understanding future market direction…or is it something that is ‘created’ by market movement.  That’s part of the research that I’m doing now.  I’m working on researching if there is any correlation / causation between twitter sentiment and price movement.  My gut tells me that there is correlation and the data points to this…but is twitter sentiment leading or lagging the market?  That’s the question (at least on of them that I have).

Is the Twitter Sentiment believable/accurate?

One of the things that has bothered me since day 1 of this research is whether the sentiment found via my twitter collection / analysis engine is ‘accurate’.   My analysis is only as accurate as my training data set…and right now, my training dataset shows an accuracy of ~90% using the Python Natural Language Toolkit’s accuracy measures.  So..I feel good there.

But…I wanted to look at comparing my Bear / Bull Sentiment Ratio with other sentiment measures to get a feel for how it might measure up.

I had a moment of brilliance yesterday.   Well…not really brilliance…more like a random thought. But…who’s to say brilliance isn’t really just randomness in the universe 🙂

I decided to compare twitter sentiment to the American Association of Individual Investors (AAII) Sentiment survey data that they release every week.

The latest AAII Sentiment Survey is shown below. I’ve taken the data from the latest survey spreadsheet and created a similar ratio to what I am using (I divide the bearish sentiment by the bullish sentiment).

Taking the AAII data and my Weekly Twitter Sentiment Bear/Bull data, I did a quick comparison between the two.  While the extremes are different, the data  generally follows similar patterns.

Not bad. The levels of the Bear/Bull ratio are different and more pronounced in the AAII survey, but the overall ‘direction’ is similar.

I feel good about this.  AAII reaches out to professional money managers for their sentiment survey…I’m watching Twitter to gather sentiment from people who are talking about the market.  While the data points are exact matches, the directional bias seems to be close.

Look for more on this in the future.

Using Twitter Sentiment for predicting stock price movement

I just finished giving a presentation titled “Will Twitter Make you a better investor?”…and like I always do with these presentations, I recorded one of my rehearsal’s to share.

In this presentation, I provide an overview of my research into using twitter sentiment and message volume as inputs into modeling stock price movements. A quick and dirty linear regression model using Twitter Sentiment, the Number of Tweets per day, the VIX Closing price and the VIX Price change delivers a simple model for the S&P 500 SPY ETF that has an accuracy of 57% over 6 months (tested on out-of sample data). This model was built using data from July 11 2011 to August 11 2011. Note: Accuracy is a measure of predicting the direction of movement.   Being accurate and making money from that accuracy is two different things.

Update:  Please note that the Linear Regression model described in this presentation is far from ideal. When modeling Time Series data, the linear regression model must be used with care due to autocorrelation issues.  

If you don’t want to listen to me yammer, you can jump down to the bottom of this post and take a look at the slides.

The presentation (if you don’t see anything…jump over to Vimeo to watch it there (~30 minutes)):

Twitter Sentiment & Investing – modeling stock price movements with twitter sentiment. from Eric D Brown on Vimeo.

The slides (if you don’t want to listen to me yammer):

Will Twitter make you a better investor?

Will Twitter Make You a Better Investor? A Look at Sentiment, User Reputation and their effect on the Stock MarketMy paper, titled Will Twitter Make You a Better Investor? A Look at Sentiment, User Reputation and their effect on the Stock Market, has been published in the Conference Proceedings for the Southern Association for Information Systems (SAIS) 2012 Conference.

You can grab a copy of the PDF here: Will Twitter Make You a Better Investor? A Look at Sentiment, User Reputation and their effect on the Stock Market

You can see the full proceedings of the SAIS 2012 Conference here.

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.