On the left, the autocorrelation of hand-classified S&P 400 tweets that have non-neutral sentiment. On the right a histogram of the calendar time intervals between those tweets. From this I'm concluding — but I'm not suprised by this — that sentiment trends. The real factor will be the cross-correlation function between net sentiment and market returns. That's going to require more data before we can examine it properly.

 

Hand classified into the set {Bullish, Bearish, Unknown}. This is to build a training set for a classifier (so I don't have to hand classify the millions that are streaming in). Here's an interesting chart of accumulated net sentiment:

 

After some playing around on the internet (see for example http://text-processing.com/demo/sentiment/) I decided I was going to have to build my own training set to classify tweets as bullish, bearish, or neutral. (The above example classifies "$AAPL going to the moon!" as "neutral" — leaving us to wonder what's positive sentiment — Jupiter?). To ease this task a built a quick application in python that links up pyodbc and tkinter to present the user with a simple dialog to classify tweets. Here it is: classify.py (2.24 kb). This one has comments!  

I have a few days capture of the tweet stream for users who mention S&P 400 stocks (the API will only accept 400 individual keywords) and do it including the $ symbol to indicate a ticker. e.g. $AAPL not AAPL. This is important to exclude ticker symbols that are subsets of regular words. Let's start by looking at the users:

This graph is a histogram of the age of the tweeter account (meaning the number of days from the account creation time to the tweet timestamp) in years and a zoom in days. We see three prominent features:

1. A sharp peak around 1 day;
2. A broad peak around 3 years; and,
3. An apparently triangular underlying distribution.

The first and last of these features are easy to explain.

For the first: twitter is a system which is plagued by spam. In a way you could think of it as a white list for spam since, as a user, when I follow a user I am telling people to send me information that they think is interesting to me. Clearly, a simple naive spambot model is to create an new account and immediately send a lot of spam tweets from it. Presumably, Twitter Inc.'s response to this is to identify the spambot account via well known techniques such as Naive Bayes Classifiers etc. and close down the account. This would lead to an excess of accounts with very short ages — the maximum age being the surveillance frequency of the Twitter anti-spambot's activity. Formally, we should do more analysis to investigate this hypothesis — but it seems fairly clear.

The third feature can be explained by the following stochastic model:

1. New users create accounts at an approximately constant rate λ;
2. Each user creates tweets at another apparently constant rate μ.

It is straight-forward to see with this dynamic that the distribution of tweeter account ages that appear in this sample will naturally be of a right triangular form, just as we see.

This leaves an apparent strong peak of users who tweet about the S&P 400 and have done so for around three years. So what happened three years ago?

 

I previously wrote about a simple experiment on twitter. I recently had several interesting conversations with Jeff Kinsey, who is the CTO of dataminr, a company that is analysing the twitter data stream. This refired my interest in using twitter in finance, and so I decided to look back at the data. The first thing to do is to get hold of the data. Twitter has a streaming api that outputs upto 1% of the data stream (unless you are a partner company, in which case you can get all). This is trivial to connect to and outputs json objects. Here's how you do it:

$ curl https://stream.twitter.com/1/statuses/filter.json?track=AAPL,IBM,ETC > archive.json 

That's so simple one fears that it couldn't be that easy, but it is. I wrote a simple python program to flatten this data for storage in a relational database, such as mysql. This is here: tweetstream.py (5.04 kb) (it's obtuse, because I never intended it to go public, maybe I should fix that).

When we look at the feed we immediately encounter several issues that will be relevant to analysis with high frequency tick data.

1. We are not partnered with twitter and so cannot access the so-called firehose api.
2. Jeff told me they (who are) have around a one second latency — apparently twitter is not yet in the Mahwah style colocation business.
3. The searched data is sampled according to an arbitary and bandwidth limiting methodology so is not a nicely designed subsample created by a statistician.
4. UTC timestamps are attached, but they only resolve to the second. This is not really an adequate timestamping method.

 

I've spent some time thinking about how to apply the insights of Compressed Sensing to the problem of finding a model to minimize the log-loss of a Bernoulli Trial. This was stimulated by the Kaggle competition bioresponse. Most participants in this competitions seem to be in the business of picking the "best random forest" recipe. As Kaggle seems to be dominated by entrenched users, I thought the only way to add value was to bring in a completely new idea. I've writtent up my thoughts and put them on the SSRN.