Firstly, firehose access — hurrah!

Secondly, Tweetminster Search (TechCrunch link) is… interesting. It’s a very hard problem to get right, measuring the sentiment of Twitter against a particular term; if the search term is “Labour”, do you search for tweets with the term “labour” in, expand the lexicon based on domain knowledge (“Government”, “Gordon Brown”), or perhaps search every tweet by a Labour MP? The methods and results seem to be in a very early stage right now, but this is something I’ve been thinking about and looking into, so cut them some slack for the rough edges. (Having said that, I will level this one criticism: as the service stands, I can’t really find anything useful out.)

Visualisation of political opinion, trend-spotting, disaster management and voting prediction are all going to become super hot over the next few months. Tweetminster Search is timely, and the mentioned API will be something definitely worth playing with; one area Tweetminster definitely adds value in is the curation of domain knowledge, i.e. maintaining a list of MPs and related Twitter accounts (news etc), and presumably caching those tweets. Firehose or no, having a readymade domain specific API is a NLP hacker’s dream. Honest.

This change, which apparently happened last Thursday along with the retweeting API and other fancy things, completely passed me by. So that’s why I’m talking about it nearly a week later. The big news? Twitter’s changed its default prompt, the question that every tweet is meant to answer, from “What are you doing?” to “What’s happening?”.

I think it’s interesting. Many tweets bear no resemblance to the ‘old’ question — conference and sporting blow-by-blow commentaries to interesting links, pieces of news and gossip, questions to the twitterverse, and random musings. Some did, of course; the almost canonical ‘eating cereal for breakfast’ and ‘in a queue behind the most annoying woman ever’ type of message, the daily commentary on one’s life that, interspersed with commentary on the wider world, is what makes Twitter so fascinating.

It’s not carefully considered and drafted news tweets or observations on the best MLM strategies that make Twitter fun, it’s the unedited stream of pure human honesty that flows from our hearts via our fingers with nary a look-in from our minds. It’s the things that annoy us, the fact that it’s wet outside, the frustration that Jedward didn’t get the boot (or the disappointment that they did). Certainly from the point of view of data-mining, heartless though it may seem, people being… well, people… is an intriguing fishbowl to glance into.

The fact that most people basically ignored the old ‘question’ means that changing it probably won’t fundamentally change Twitter. It more mirrors, rather than propels, a shift in the way Twitter is being used by citizen journalists and commentators the world over — and an attempt to get away from the dogged old ‘breakfast’ use-case that even I trot out time and again. Maybe it will make people stop and think a little when they’re about to post some banality or other, though, and that saddens me just a little.

Edit: It’s also interesting that Facebook’s question is “What’s on your mind?”, staying me-centric; this reflects the difference between the two services rather well, I think.

Time-tracking is a really cool thing to do. Why? Because we have faulty memories, and we like monitoring and planning. Nobody can reasonably be expected to remember by 5pm on Friday what they were doing on Monday afternoon. But knowing where your time over the week goes is invaluable, whether you’re a run-of-the-mill employee, an entrepreneur, or a freelancer juggling clients.

There are some gorgeous time-tracking solutions out there, yet I personally just have an allergy to typing stuff into a web app.

So this is what hit me last night, at 2am, embroiled amidst caffeinated insomniac thoughts of hair dye and giraffes: why isn’t there a Twitter time-tracking app?

Maybe there is. In fact, I hope there is, because I want to use it. Lazyweb?

In case there isn’t, and someone’s out there looking for something to build (hey, that ’someone’ could be myself in a few months’ time.. who knows):

Let me constantly microblog what I’m doing, in an enterprise context, on a private level, so I can look back and figure out what I’ve done. Use hashtags or another way of formatting keywords to mark out specific types of task and use some simple natural language understanding to automatically graph and plot my time.

Aha! A bit of Googling later and I find Tempo and Twistory. Both potential solutions… but without the latter ‘intelligent’ part, in a way.

The problem with all this is it does require discipline. You gotta tell Twitter, or whoever, what you’re doing. Plus, as you can only go back so far with tweets, I’d suggest setting up an API script to archive your tweets at close of work on Friday (or Saturday, or Sunday…). And yet, the advantage of using a fairly free-form entry method – one that’s close at hand, too – and building your own intelligence around it is you can add in extras, like an end-of-day mood summary, comments, notes, etc. Maybe step 1 is to start tracking now, and step 2 to build in the AI later…

Voting has traditionally been a wondrous mish-mash of politics and geography combined with points directly proportional to the cheesiness of the act. For example, the UK always tends to vote Ireland up, and vice versa; Eastern European countries pat each other on the back, and Germany never gives points to France.

(This is somewhat of an exaggeration, but as a teenager, Eurovision was how I learnt international politics and, later, the French for ‘Bosnia-Herzegovina’.)

So, I went on what I’ll fondly call a public transport experiment on my way home from the airport on Saturday night. This is relevant, because it means I was on a bus in the middle of nowhere for most of Eurovision. Fortunately, thanks to Twitter, it was as if I was sat at home in front of the TV.

Nothing really comes close to Twitter for event coverage when you’re away from civilisation. It really was amazing. Snark and sarcasm from celebrities coupled with genuine patriotism, descriptions of astounding costumes, and mildly-concealed insults (it’s not xenophobia if it’s Eurovision, right?).

The First Eurovision Problem

The title of this post is misleading; there are two Eurovision problems (discounting the fact the UK didn’t come last, disappointingly).

Firstly was my simple inability, when on the move, to only follow certain Eurovision-related tweets. I heard that @Schofe and @Wossy were providing great commentary, but their tweets either got lost in the flood of ‘all updates’ or ‘all eurovision’; I didn’t have a way to see ‘all (friends + eurovision)’.

Nor did I, using Tweetie, have a way to temporarily define a group of people whose updates I wanted to follow. I was tempted to create a new Twitter account just to follow a few people and get Eurovision that way, but figured it would be too awkward to do this by phone.

Of course, this is all my own fault for following so many people in the first place, so I suppose the solution would be to do a grand Twitter prune, or set up a second account just for information overload. But that doesn’t really seem in the spirit of it.

The Second Eurovision Problem

This is a fun and meaty information filtering problem that relates to realtime predictions in a big way. I didn’t have a chance to watch Hubdub/Betfair/etc change as the show was going on, but I dearly wish I had.

Clearly, as people see the various acts, their opinion of the best one changes. Thus the probability of a certain act winning changes over time as more variables enter the equation. This is also affected by hype and, sadly, the aforementioned geography and politics (although I think this is less the case than it used to be).

With Eurovision, it’s likely a safe bet to say that as each act plays, it introduces a new probability of that act winning into the overall picture, and also affects the probability of previous acts’ victories. (Note that a bad song may increase the previous acts’ chances!)

The probabilistic question is whether to start off assuming each act is equally likely to win, or to break time into discrete units and assume that only acts that have played so far have a probability of winning (so at t=2, with two countries having played, the only possible winners are those countries).

Perhaps a mix of the two, mirroring the viewer’s tendency to ‘pick a favourite’ but also look forward to certain new acts. This combines hype and visibility. Once the act has played, it becomes a known variable, affected by future acts but also far more tangible than before.

Would you feel more or less comfortable putting your money on Norway before or after they have played? How about after everyone has played? At what point would you commit £100 to a win – or would you always hedge and put some on your second favourite?

Where this becomes a really interesting problem, for me, is in social media analysis. I was very tuned into the Twitter conversation around Eurovision, although due to information overload and 3G black holes I didn’t see or digest every single tweet. What took part was the pub or living room conversation, on a larger scale.

To what extent did Twitter sentiment about the Eurovision participants reflect the overall voting?

To what extent did it reflect the voting of the United Kingdom?

To what extent was it wildly wrong?

The latter is interesting. Given country X, with a ridiculous Euro-trash entry in some language nobody’s ever heard of, with pink hotpants and glitter and other ridicule-worthy aspects, the conversation traffic about it might be surprisingly positive. It would certainly be disproportionately high given the entry’s quality.

But does this reflect perhaps a sympathy vote? If everyone’s ridiculing Nowherezikstan, does that stop at Twitter snark or does it translate into points? How can we tell the difference between genuine excitement, ridicule just because it’s bad, and ridicule because it’s so bad it’s actually quite good?

Back to the first two questions. Thanks to Twitter geocoding, we can strip out the UK opinion from everyone else’s, or we can just assume that the majority of English-speaking tweets who care about Eurovision will come from the UK. We do need to do some filtering, or else we will just assume our own country wins; as countries can’t vote for themselves, we need to remove that as a possibility.

The ultimate question and gold standard involve two things: how do the betting companies do it? and how can we build something that reflects twitter/online sentiment (think Facebook Connect on a Eurovision live stream) over time, comparing that to votes? It’s like a constant, ongoing, realtime poll that could affect betting as well as simply being a fun way of automatically watching bar charts change as you talk.

Of course, there are problems associated with the IR/NLP side of things. How do we know which entry a tweet refers to? How do we track @-conversations to measure agreement with sentiment? (e.g. @Wossy says Norway’s act is amazing and 100 people say “@Wossy I agree!!!!”). How do we strip out the sarcasm, or do we? Do we build a probability model specific to Eurovision and refine it after every act by looking at the sentiment, or do we simply track mentions and normalise? Do we even normalise?

There are answers to some of these problems, varying from the complicated to the simple (“We don’t”). Some of it is more experimental, to see what’s the best result. And some of it is just academic fun :)

So, next year, if you see an interactive, realtime, constantly-changing chart of who’s going to win Eurovision, you know who created it — and some of the hurdles along the way!

Since the start, you could view your Twitter stream in three ways – server-side, so this affected whichever method you used to browse Twitter. The options were to only show ‘broadcast’ messages, i.e. nothing starting with an @; to show @-replies between people you were following, i.e. only conversations where you knew both parties (the default); and to show all messages, including those @-replies directed at people you didn’t follow.

Quite sensibly, Twitter looked at the user behaviour – almost everyone kept to the default. Having tried out the ‘firehose’, ‘everyone’s tweets to anyone’ approach I’m not that surprised. Even when I was following around 100 people, my stream became vastly noisy and unmanageable. The idea of being able to discover new people through seeing half their conversations was nice, but without any client that can pull in the rest of the conversation, it was like overhearing a phone conversation on the train — annoying and unnecessary.

Product management decision ahoy: ‘let’s remove the option, nobody (<2%) uses it and keeping it there costs money without translating that cost into value’.

Enter TechCrunch, stage left, and suddenly — apart from a bit of whining that TC should shut up about Twitter, which is fair enough — it’s cool to demand the ‘firehose’ of @-replies back. The angle that ‘we’re not smart enough for it’ was very clever, and also downright underhand. Spinning a product management decision so that most of the product’s users, who were previously unaware of the feature’s very existence, now demand for it back… I guess it causes pageviews, but it’s simply ridiculous.

There’s a bit of a terminology hiccup which isn’t helping, in that people are getting the idea that their actual ability to reply to people they don’t follow will be hindered – it won’t – or that they won’t see replies from people who don’t follow them – er, no. Even the discovery aspect isn’t really a big problem, as there are plenty of conversations/RTs/etc that don’t start with an @ and introduce the username later in the text. Those’ll still show up.

Twitter will almost certainly have to reverse the change, and those complaining loudest about it and yelling ‘#fixreplies‘ from the rooftops will go back to not using it. In fact, I wonder if the canniest thing they could do right now is to put the @-firehose option on – the one that everyone’s complaining about missing out on – and watch the majority of users drown in confusion for an hour or two.

I just read an interesting article on the RT wires about why Twitter’s the future of search. A statement that initially got a nod of the head, until I started thinking in a little more detail about the arguments. I think it’s really important here to actually talk a bit about what search is.

@Gyutae’s article seems to simply equate search with ‘finding information’, but there’s a slightly deeper dimension: you want to get all the information, or the most relevant information, or unbiased information, or…

Anyway, it’s not just about finding stuff, but about the quality and source of the stuff you find.

So, six reasons why Twitter isn’t the future of search:

Social isn’t representative

Asking Twitter for an opinion is all very well, but bear in mind you are getting the Twitterverse’s opinion, not everyone. Although Twitter is becoming more ‘mainstream’, you’re still looking at a certain type of user, in a somewhat self-selecting crowd.

If you’re after the best restaurants in New York, you’re likely to get a decent cross-section of Twitter replying, but if I’m looking for recommendations for a nail salon in Birmingham and nobody’s mentioned one on Twitter, I have to poll my own network. Which is great, if I have access to the sorts of people who would know. Otherwise I have to seek out a few likely people and @ them the question, wait for replies (if any), etc. A lot of work.

In short: Search queries that don’t match the Twitter userbase don’t get good answers.

Anti-information overload isn’t always informative

Sometimes you’re simply not searching for something that can be answered in 140 characters. Sure, Twitter encourages people to be concise with their information, but if I’m after a fairly detailed explanation of something – or a howto, or a tutorial – I won’t find that on Twitter. If someone’s tweeted a link with the appropriate text, I might find it, but Twitter just isn’t the platform to search for detail on.

Realtime makes overviews hard to find

Realtime search is great for realtime applications, such as finding out the exact response to an ad that just ran during the Super Bowl, or the latest football score. But if you want historical information as well as ‘the latest’, or an overview of an event rather than the blow-by-blow tweets, you get totally overloaded.

For example, digging through #sxsw tweets to find informative nuggets was just a nightmare. Realtime search definitely has its place, but it won’t ever be the only way we search.

It’s hard to pick out accuracy from the masses

This ties in a little with #1, in that ‘the masses’ is actually ‘the masses who use Twitter’. A level playing field is great, but the advantage of something like PageRank is that you do gain an idea of how respected, influential, popular, accurate, etc. a web page is — generally people linking to it are giving it a silent ‘thumbs up’, pushing its PageRank higher.

That’s just not there on Twitter, and for various search tasks, you actually want that sort of ranking and relevance, rather than just a mass of voices all shouting at once.

Direct contact with sources isn’t always the answer

If you had a question about what it’s like to be a comic, sending Stephen Fry an @ might get you a nice 140-character answer. But if you were doing biographical research, or wanted to ask any sort of question requiring a detailed answer, or actually have an in-depth conversation, you wouldn’t use Twitter search.

Leaving aside the fact that some Twitter celebrity accounts have been known to be fake, how much value from asking someone directly can you really get, compared to reading published information about them?

On the flipside, if you have a question that suits a very specific person – maybe not a celebrity, but how about an entrepreneurial mum from Wisconsin? – you can find that person from Twitter, whereas you’d be lost on a more conventional search engine (until you find WorkAtHomeMomsFromWisconsin.com, of course).

I’m not saying that this level of trust and source interaction is bad, but it’s not ‘the future of search’.

Location awareness is unreliable

Using just the location associated with a tweet and saying ‘every piece of content from this location is related to it’ is just plain silly. A lot of Twitter conversation is location-free, and the only real application of this is to resolve statements like ‘Back from Starbucks. Wow, what nice service!’ to mean ‘Starbucks in Edinburgh has nice service’ because Twitter knows I’m in Edinburgh. A lot of statements posted from a location talk about other locations, even.

Having some form of location-knowledge about a person is great, but it’s got to overcome some serious hurdles before it can accurately be used in search. However, it does make finding the aforementioned work at home mother easier, and location definitely is part of the future of the Web.

If you liked this post, why not tweet it?

So that’s the problem: how do I turn off discussion of an event I’m not at, without unfollowing people I like (and who might still make interesting comments outwith the hashtag)?

Filter Tweetdeck to exclude, not include

I’ve just poked around in TweetDeck a little, and the simplest thing to do is just filter your friends. You can either put the people you want to listen to in a specific group, regardless of what they’re saying, or add a filter to All Tweets to remove a keyword. Sorted.

This is a fairly quick-and-dirty (I hesitate to say inelegant, since the simplicity is elegant in and of itself) hack to blinker yourself. I don’t like it because some useful stuff will, inevitably, pop out of a widely-attended and widely-tweeted event.

Picking out the gems

Let’s use that wonderful wisdom of crowds again, and use a couple of measures to filter tweets from an event (let’s use the highly topical #sxsw), here presented in terrible pseudo-rule format:

• tweet default SHOW
• If (tweet contains #sxsw) default IGNORE
• If (tweet contains #sxsw) AND (tweet contains http://) default SHOW
• If (tweet contains #sxsw) AND (tweet contains RT) default SHOW
• If (tweet contains #sxsw) AND (tweet sentiment = very positive) default SHOW
• If (tweet contains #sxsw) AND (tweet sentiment = very negative) default SHOW

More sophistication:

• foreach (tweet contains http://) IN (tweet contains #sxsw):
  url_frequency[tweet.re((http://[a-zA-Z0-9._-/]))]++
  url_frequency.trim_sparse(5) // function to ignore anything appearing less than 5 times
  print url_frequency.keys()
• foreach (tweet contains RT) IN (tweet contains #sxsw):
  RT_frequency[tweet]++
  RT_frequency.trim_sparse(5)
  print RT_frequency.keys()

Sorry, that code is pretty horrible, but I hope the meaning is clear enough! Basically, show any RT’d message or URL that is repeated more than five times overall.

Righteous justification for being absent

Not everyone’s absent from an event like SXSW by choice; some of us are limited by factors such as time, money, being on the wrong continent, etc. As such, we get happily righteous when it turns out that those at the event are having a stonkin’ miserable time (example: the recent tweets from Brits in Austin about the terrible weather. Hurrah! cry those of us at home.)

So, easy enough to do: http://wishyouwerenthere.com, a waterfall tweet display that takes a keyword and uses the magic of AJAX to show people exactly what they’re not missing by picking out all the horribly biased, negative tweets about it. Who cares about the positive when you’re not taking part? Exactly. Now to see if I can code that up tonight (feel free to beat me to it, it’s a busy weekend what with BarCampScotland tomorrow and some application deadlines looming)!

Also, finally, I promise I’ll post something that isn’t about Twitter soon. Honest.

I was playing around with some natural language experiments today and stumbled across a quick and dirty idea, which I’ve spent the last half an hour implementing. It’s so easy to realise stuff right now thanks to APIs and toolkits and Python slowly becoming my second language.

@GenTweets is word prediction brought to Twitter. Given one word, we compute the probabilities of all possible (seen) following words, and then choose one, repeating until we have a complete tweet. It’s pretty naively implemented at the moment and I’m already thinking of things to do (involving, but not limited to: ignoring non-English tweets, better punctuation handling, allowing interaction, allowing a dynamic start word — currently it’s set on ‘Today’ — and using more exciting ways of picking the right words. Plus there’s data visualisation, different time windows, hashtag-, user- or topic- based tweet indexing, etc.)

My immediate urge is to do a SXSW one that can generate random SXSW-related babble that will be nearly indistinguishable from the real babble going on over the weekend. Jealous, moi? I’m also thinking about other ways of helping attendees cut through the noise — based on recent tweeted events I’ve been to, there will be a lot of #sxsw-tagged tweets, and how can I help people deal with them?

It’s every CEO’s nightmare: waking up to find mails and tweets pointing out a new competitor, someone who — at first glance — appears to have pipped you to a particularly juicy post. Thus began my morning when I was alerted to Twendz, a new sentiment-based Twitter monitoring app.

However, I actually think Twendz is a good sign. To be boisterously arrogant, the app itself doesn’t hope to compete with what I’m working on in terms of technology, as it’s using the most naive of sentiment classification techniques. (OK, so it could incorporate more advanced classification, which might get worrying.) It also doesn’t really show any useful information — sure, you can get a barometer based on the last three hours or so, and there’s a jolly pretty waterfall of tweets, and the tag cloud is a nice touch, but beyond that? It’s candy, not something you can sit and digest and act on.

The fact it’s been launched by a PR firm is nothing but good, though. It’s a clear sign that at least one firm acknowledges the value of sentiment and opinions in social media, and sees a need for better ways to search and track them. It’ll be interesting to see what sort of press the app gets and it’s certainly alerted me to something that was hit home quite squarely yesterday: think small, not monolithic, release early, and see what happens. If I’d coded this up months ago, which I certainly could have done, who’d be getting the press now? Lesson learned.

Taking the notion of a user-powered brand to a new level, Skittles has been the talk of the town today as its website changed to consist of a ‘new meeja’ web 2.0 box-ticking mashup. Well, not really a mashup so to speak; the basic homepage is a Twitter search, with links to the company’s Wikipedia, Youtube, Flickr and Facebook pages.

It’s great to see a company that’s selling a fairly old-school product start to jump on the social media bandwagon. Although it can easily backfire, with clued-in Twitterers posting ribald comments right left and centre, I’d guess that the social media traffic about Skittles for just today probably outweighs the whole of last year’s. After all, who tweets that they’re eating a packet of sweets? (Oh, wait.)

It looks like the end result of this is volume rather than quality of conversation. Everyone’s tastebuds are different, so a wave of people saying they don’t like Skittles (in fact, I don’t!) won’t really harm the overall perception of the product. The more dubious comments about cancer etc will mostly be taken with a pinch of salt, although the joy of the Internet is that even a side comment can explode into a meme and become entrenched; a tweet could, in theory, destroy Skittles (as the #skittlefisting tag is trying to).

What’s interesting to me is visualising the reaction to this change: looking at the proliferation of posts throughout the blogosphere, as well as the change in mood and tone of Skittles tweets over time. When did people start trying to game the system? When did (or will) they lose interest? How does the volume of chatter correlate with Facebook fan page subscriptions, YouTube views, Wikipedia edits? And how do these numbers relate to sales? Pretty cool to think about, especially as a case study. Apparently Omniture is providing analytics, so I’m especially keen to investigate what they deliver and where the gaps in their service are.