TextWise

In my last post, I presented some research on the different content types we found in our corpus of 8.9 million Twitter messages. One surprising result we found is that Portuguese is apparently the second most common language on Twitter, beating out both Japanese and Spanish. Given the unreliability of TextCat on short pieces of text, I decided to verify our language statistics by looking at the location field in the user info for the unique set of users in our corpus. This was not a straightforward thing to do, however, because the location is a text field which people can write absolutely anything they want into. For example, the following all occurred more than once in our corpus:

• “New York”
• “NYC”
• “everywhere!!!!”
• “In ur computers, eating ur RAM”
• “Earth”
• “Mars”
• “Utah :)”
• “utah :(”

To get around this problem, I normalized the text by converting it to lowercase, removing punctuation, and changing things that looked like addresses to have just the city (so that “123 Fake St., Springfield, USA” becomes just “springfield”). I then looked at the top 500 locations in terms of number of twitterers. These are the most common countries represented in users’ locations:

And the top 10 cities are:

1. New York
2. São Paulo
3. Los Angeles
4. London
5. Chicago
6. San Francisco
7. Rio de Janeiro
8. Tokyo
9. Atlanta
10. Toronto

While the locations are dominated by English-speaking countries, Brazil does come in second in terms of number of users, and two Brazilian cities show up in the top 10, which suggests that our language stats aren’t too far off the mark.

Another question we considered in our study is whether there is any way to distinguish between twitterers who post broadly informative messages from those who post mainly personal messages or spam. Our first thought was that the number of followers a twitterer has would be a good indication of how informative their messages are to a wider audience. But we were quite surprised when we looked at the distribution of the number of followers in our sample:

The x-axis here is the logarithm base 10 of the number of followers. While most twitterers in our corpus have between 15 to 60 followers (log=1.2 to 1.8), there is a long tail where we can find accounts with more than a thousand, 100,000, or even a million followers. We didn’t realize at first the number of celebrities currently using Twitter, as you can see in this list of the top 100 most-followed Twitter accounts. Of course, it’s a matter of opinion whether the latest funny video that Ashton Kutcher found on YouTube is more important than what Barack Obama has to say about health care, but for our purposes, we’d rather filter out celebrity ramblings from the more serious messages, and that is not easy to do based on the number of followers alone.

A more surprising fact we discovered is that spammer accounts can have relatively high numbers of followers as well, as you can see in the following boxplot:

This data is from the 1,000 tweet sample which was classified by message type that I discussed in my previous post. In this plot, messages about the user’s current status and private conversations are grouped together as “personal” messages, while all other messages (excluding spam) are “info” messages. The boxes show the middle 50% of the distribution for each type, while the whiskers extending from the boxes show where 99% of the data points lie. (There are a few outliers above 5,000 followers which are not shown here, to make the distributions easier to see.) While spam messages only made up a small fraction (4%) of our sample, the plot shows that within the set of spammer accounts there are quite a few which have more than 500-1000 followers, a number which would be pretty high for the other two message types. There is even one spam account in our sample which had over 10,000 followers at the time they posted.

But how could a spammer get so many followers, given that all they post is spam? Given that for nearly all of these accounts, the number of friends (accounts they are following) is greater than the number of followers, I suspect that what’s going on is that spammers go around following other twitterers at random, and at least some of these people are following them back out of courtesy, without realizing that they are actually a spam account. The only way a Twitter spammer could get someone to see their tweets is if they are followed by them, after all. There’s probably a high rate of turnover in a spammer’s followers list, but that wouldn’t matter much, as long as they can find more people to follow who will follow them in return without checking them out first.

All of this means that distinguishing spam from informative tweets will not be easy, even if there isn’t that much of it currently. But some good news for us is that twitterers who post lots of informative content do tend to have more followers than those who post mainly personal messages. This fact, combined with some semantic analysis of Twitter messages, should help us a great deal in mining the Twitter stream for useful content.

Recently we’ve been looking at how well our Semantic Signatures technology works with messages posted to Twitter. These kinds of messages pose significant challenges for the semantic web in general, because their extremely short length (140 characters or less) means that there will be very little context available for understanding the content of the message. In addition, many of these messages feature “creative” spellings and grammar, and are of a personal nature (e.g. “Having sushi for lunch today”) that would not be of general interest. Extracting any meaningful information from these snippets of random conversation will be quite a difficult task indeed.

To see what exactly we’re up against, we undertook a small study to characterize the different types of messages that can be found on Twitter. Read the rest of this entry »