In my last post, I presented some research on the different content types we found in our corpus of 8.9 million Twitter messages. One surprising result we found is that Portuguese is apparently the second most common language on Twitter, beating out both Japanese and Spanish. Given the unreliability of TextCat on short pieces of text, I decided to verify our language statistics by looking at the location field in the user info for the unique set of users in our corpus. This was not a straightforward thing to do, however, because the location is a text field which people can write absolutely anything they want into. For example, the following all occurred more than once in our corpus:

• “New York”
• “NYC”
• “everywhere!!!!”
• “In ur computers, eating ur RAM”
• “Earth”
• “Mars”
• “Utah :)”
• “utah :(”

To get around this problem, I normalized the text by converting it to lowercase, removing punctuation, and changing things that looked like addresses to have just the city (so that “123 Fake St., Springfield, USA” becomes just “springfield”). I then looked at the top 500 locations in terms of number of twitterers. These are the most common countries represented in users’ locations:

And the top 10 cities are:

1. New York
2. São Paulo
3. Los Angeles
4. London
5. Chicago
6. San Francisco
7. Rio de Janeiro
8. Tokyo
9. Atlanta
10. Toronto

While the locations are dominated by English-speaking countries, Brazil does come in second in terms of number of users, and two Brazilian cities show up in the top 10, which suggests that our language stats aren’t too far off the mark.

Another question we considered in our study is whether there is any way to distinguish between twitterers who post broadly informative messages from those who post mainly personal messages or spam. Our first thought was that the number of followers a twitterer has would be a good indication of how informative their messages are to a wider audience. But we were quite surprised when we looked at the distribution of the number of followers in our sample:

The x-axis here is the logarithm base 10 of the number of followers. While most twitterers in our corpus have between 15 to 60 followers (log=1.2 to 1.8), there is a long tail where we can find accounts with more than a thousand, 100,000, or even a million followers. We didn’t realize at first the number of celebrities currently using Twitter, as you can see in this list of the top 100 most-followed Twitter accounts. Of course, it’s a matter of opinion whether the latest funny video that Ashton Kutcher found on YouTube is more important than what Barack Obama has to say about health care, but for our purposes, we’d rather filter out celebrity ramblings from the more serious messages, and that is not easy to do based on the number of followers alone.

A more surprising fact we discovered is that spammer accounts can have relatively high numbers of followers as well, as you can see in the following boxplot:

This data is from the 1,000 tweet sample which was classified by message type that I discussed in my previous post. In this plot, messages about the user’s current status and private conversations are grouped together as “personal” messages, while all other messages (excluding spam) are “info” messages. The boxes show the middle 50% of the distribution for each type, while the whiskers extending from the boxes show where 99% of the data points lie. (There are a few outliers above 5,000 followers which are not shown here, to make the distributions easier to see.) While spam messages only made up a small fraction (4%) of our sample, the plot shows that within the set of spammer accounts there are quite a few which have more than 500-1000 followers, a number which would be pretty high for the other two message types. There is even one spam account in our sample which had over 10,000 followers at the time they posted.

But how could a spammer get so many followers, given that all they post is spam? Given that for nearly all of these accounts, the number of friends (accounts they are following) is greater than the number of followers, I suspect that what’s going on is that spammers go around following other twitterers at random, and at least some of these people are following them back out of courtesy, without realizing that they are actually a spam account. The only way a Twitter spammer could get someone to see their tweets is if they are followed by them, after all. There’s probably a high rate of turnover in a spammer’s followers list, but that wouldn’t matter much, as long as they can find more people to follow who will follow them in return without checking them out first.

All of this means that distinguishing spam from informative tweets will not be easy, even if there isn’t that much of it currently. But some good news for us is that twitterers who post lots of informative content do tend to have more followers than those who post mainly personal messages. This fact, combined with some semantic analysis of Twitter messages, should help us a great deal in mining the Twitter stream for useful content.

Recently we’ve been looking at how well our Semantic Signatures technology works with messages posted to Twitter. These kinds of messages pose significant challenges for the semantic web in general, because their extremely short length (140 characters or less) means that there will be very little context available for understanding the content of the message. In addition, many of these messages feature “creative” spellings and grammar, and are of a personal nature (e.g. “Having sushi for lunch today”) that would not be of general interest. Extracting any meaningful information from these snippets of random conversation will be quite a difficult task indeed.

To see what exactly we’re up against, we undertook a small study to characterize the different types of messages that can be found on Twitter. We downloaded a sample of tweets over a two-week period using the Twitter streaming API. This resulted in a corpus of 8.9 million messages (“tweets”) posted by 2.6 million unique users. About 2.7 million of these tweets, or 31%, were replies to a tweet posted by another user, while half a million (6%) were retweets. Almost 2 million (22%) of the messages contained a URL.

Next we used a modified version of the language-guessing program TextCat to find the distribution of languages in our corpus.

As you might expect, English turns out to be the most common language used on Twitter (61%). But surprisingly, the next most common language is Portuguese (11%), beating out both Japanese (6%) and Spanish (4%). This is the opposite ordering of these three languages compared to what we find on the internet in general (http://www.internetworldstats.com/stats7.htm). It seems that Twitter must be a lot more popular in Brazil than in the rest of Latin America. Other languages like French, German, and even Malay are also fairly common in our corpus. It’s hard to get accurate counts for these, though, because TextCat doesn’t deal well with shorter texts (note the estimated 10% of messages that are unknown or misclassified), and it doesn’t seem to be able to identify these languages in our sample as reliably as the first four languages.

We were then interested in seeing what kinds of messages get posted to Twitter. Is it really all just people talking about where they are right now and what they’re having for lunch, or is there actually some informative content out there for us to find? To answer this question, we took a random sample of 1,000 English-language tweets from the corpus we collected, and then classified each message as one of the following types:

• User’s current status — where the user is right now, what they’re doing, etc.
• Private conversations — some twitterers seem to use the service as if it were a giant internet chatroom
• Links to web content — a URL with an article title and/or some commentary on its content. Further broken down into: links to blog and news articles; links to images and videos; and other links.
• Politics, sports, current events — discussion of these topics
• Product recommendations/complaints — recommendations or complaints about specific TV shows, movies, techie gadgets, etc.
• Advertising — posted from a company’s twitter account
• Spam — a strange phenomenon, given that an account has to be followed for anyone to see its tweets, but it does exist
• Other messages — messages that don’t quite fit under any of the above categories. Fan messages to celebrities, shoutouts to other users, web-based polls and quizzes, and so on.

Here is a graph showing the frequency of these different message types in the 1,000 tweet sample.

As you can see, over half of the tweets in the sample are either user statuses or private conversations. Only about 10-20% of the messages could be considered more broadly relevant to a larger audience (depending on which message types are considered to be informative). So while there is some interesting content to be discovered on Twitter, it will definitely take a bit of work to find it.

We also collected some statistics on the twitterers in our sample, such as their average rate of posting, the most common clients they use, and so on. But this post is already pretty long, so I’ll save that for a later date. Stay tuned…