With the immense prevalence of Twitter apps, hacks and mashups out there, the concept of a new idea is fast becoming a fallacy. However, I have a problem, and I’ve not yet found a handy solution.

This particular problem, as alluded to here and with various Tweets I’ve sent, is one of catching up. Before the anti-Twitter brigade get going, let’s be clear here: this isn’t just a Twitter-related problem. I subscribe to users on FriendFeed who post items there, and 95% of my Facebook friends post everything on Facebook; I’ve also got a (long unused) set of Livejournal contacts who routinely post interesting stuff, not to mention the number of blogs and del.icio.us feeds I subscribe to and don’t bother reading.

This is a slightly different problem to the one the professional blogger faces — my motivations for reading this stuff are mostly personal, although as one of today’s digital nomads there’s a heavy professional element as well. It isn’t the end of the world if I miss a posted item or two, and I’m fairly interested in the chatter between friends as well as the meat of what they post, hyperlinking themselves into the World Wide Web.

Still. Let’s distill this problem down a little. First off: my friends post too much stuff for me to reasonably attempt to keep up with it using the original services’ interfaces. Second off: I care what my friends find interesting. Third off: I like having a ton of links to open first thing in the morning over coffee. Things that will enrich me, make me smile, interest me, teach me, or inspire me.

How do I currently deal with this? Well, how does anybody? We all have different approaches to this personal level of information overload (oh, how I am coming to detest that phrase).

Step 1 – Cut the fluff

I don’t read things I don’t need to. My Facebook stream is easily caught up with by logging in once a day, or every other day. The level of content there is fun and usually worth reading – finding out what my friends and colleagues are up to does have its worth! – but it’s not going to kill me if I don’t find out about it right this minute.

Similarly, the way I personally use FriendFeed leads to a lot of duplicated content (Tweeted links from people I already follow on Twitter, for example) and it’s really only useful for me because interesting stuff from friends of friends floats up. Your mileage probably varies; I’m by no means a powerful FriendFeed user, as I haven’t quite got the point of it yet, though I like some of the concepts it uses. Either way, I get by checking my FriendFeed purely from the single email a day.

Talking of email, I filter my mailing lists quasi-religiously and use Gmail’s multiple inboxes (in Labs) so I can more easily ignore the volume of email coming through certain labels. I can catch up with these at my leisure; they’re not clamouring for my attention 24/7. But due to the multi inbox placing, I don’t forget about them either.

Twitter is the biggest source of noise for me, and I do a wee bit of filtering on this by not following people whose tweets simply don’t interest me. However, a lot of people post a lot of cool stuff, so my follow list — small though it might seem — is still pretty loud and leaves a lot of catching up to do in the morning since I have the misfortune to be 5-8 hours ahead of half my subscriptions.

RSS is something of an interesting one. I subscribe to loads of RSS feeds, but I’ve stopped reading them. When I have downtime I load up my iPhone and catch up with my ‘key’ feeds – others I subscribe to on Twitter. The rest of them I download every couple of days and save for long journeys so I can look busy when there’s no ‘net connection. I don’t take long journeys often. I used to be a total RSS junkie; I find I get the same level of ‘cool stuff’ out of the other networks I tap into that I used to get out of RSS. However, there are still a couple of feeds from particular people that I check by manually visiting their blogs every few days.

Step 2 – Filter the rest

Okay, so I’ve already cut out a lot of the ‘problem’ by completely ignoring the realtime web and checking things at my own leisure. But I still have Twitter jumping up and down crying for attention in the corner.

This is where I evangelise TweetDeck, as so many do. Three key features: groups, filter, and search.

Groups allow me to set up columns of people I want to read more than others. It’s almost an inverse of my Gmail mailbox, where I cluster and separate stuff I don’t want to read. Unless I’ve totally missed something, when you add a new follow you have to manually add them to relevant groups, so I try to keep my groups organised to save me this effort; most of them won’t grow. I use groupings like ‘RL friends’, ‘Edinburgh people’, ‘Bloggers’ and ‘RSS bots’, although due to reinstalling recently the only group I currently have set up differentiates people I’ve known and followed for a long time from people I just started following. As I follow more people, my stream gets polluted with stuff that’s less and less relevant, so cutting that out helps with the headache.

I still spend a lot of time reading All Tweets, though, and this is where Filter is quite fun. Try filtering your tweets column with the term ‘http’. Instantly you just get the day’s links to peruse at your leisure. Good stuff.

Search is also cool, although it can be overwhelming. It’s just a shortcut to search.twitter.com which allows you to keep everything within TweetDeck; I used it to track the #edtwestival, and would’ve used it for the conferences I was at, had it existed back then.

Step 3 – Pipes, RSS and mashups

This is where it all gets a little kooky and theoretical. Here’s the deal: Twitter provides your friends’ updates via RSS, as well as via its API. Blogs, del.icio.us and poor abandoned Livejournal also have handy RSS feeds. A quick search implies you can get at least some FriendFeed content via RSS. Facebook? Maybe not, but I already decided Facebook content can wait. Still, there’s a theme here.

Many RSS feeds plus the desire to do something cool to them? Enter Yahoo Pipes, as inspired by this Twitter-Pipes URL extractor mashup. (Aside: I’m pretty impressed by the stuff Yahoo is offering for mashups and Internet tomfoolery. BOSS, Pipes and Term Extractor are all fun to play with.)

Using Pipes, it’s theoretically possible to amalgamate a dozen or more RSS feeds of your choosing, strip out all but the relevant http:// links and display them in a handy ‘Here’s your link bag for the night, madam’ format. (I’m imagining Stephen Fry presenting me a stack of virtual newspapers on a virtual silver platter. Too far?)

This is where it gets a little problematic. Firstly, my brief experimentation with Pipes involved having to call a URL of the format http://username:password@twitter.com/statuses/… in order to get my friends timeline. The geek in me won’t touch cleartext with a bargepole. Secondly, I don’t really think this is intelligent enough to produce the level of results I want. I don’t think it’s too hard to beef it up a little, though.

Here’s the deal. Using the API, or RSS, or messenger pigeons, show me: the URLs tweeted by my Twitter friends in the last 24 hours, sorted by interestingness and longURL’d to prevent duplicates. Let me add in other RSS feeds that commonly contain links such as ‘best of’ blogs, delicious feeds, etc. Once I open them up, give me some way of tracking where they originally came from, so I don’t end up confused and unable to attribute if I retweet.

It seems to me that mashing up a delicious or similar service on top of this URL extractor would work really nicely. (Why roll your own when you can overload someone else’s?). When someone tweets a link it gets added to this bucket o’ URLs with tags extracted from the tweet itself, and metadata indicating its source. When I go to retweet, that info is saved, so my retweet writes itself (of course, I can edit it if I really don’t want to give the source credit). Either automatically or manually, I bookmark/star/’like’ items that appeal to me and my friends do the same, thus giving the ‘interestingness’ ranking I handwaved about a paragraph earlier. Perhaps everything I click on gets +1 to interesting, and stuff I explicitly like gets +5. If n is the number of times tweeted/retweeted, interesting = n (clicks + 5 like).

I would also be able to mimic some popular tweet-universe services such as retweet tracking and URL popularisation but to keep them within my friends cloud. What’s the point of limiting my information sources if I don’t take advantage of those limits? Of course, I care about what The Internet thinks (I wouldn’t be in this business if I didn’t), but I also want to know what people I respect and listen to think.

If I ever get a spare weekend, I’ll code this up. If someone’s already beaten me to it, all the better!

I’ve been spending a lot of time playing about with the Twitter API recently, putting together a couple of quick projects which are partially powered by sentiment AI and partially just fun(tm). Along my wanders I appear to be accruing Twitter followers at a rate that, while modest by comparison to most people, means my time spent monitoring the platform and simply engaging in meta activity (checking out new followers etc) is growing.

It gets to the point where, with an unfiltered stream-of-twitter-consciousness, I spend far too long first thing in the morning just catching up on what people have said while I was asleep (a side effect of following Americans and insomniacs). It’s a bit like reading vast numbers of RSS feeds (another habit of mine); my eyes glaze over halfway through and I barely skim read most of the messages, although fortunately due to years of top-notch academic training I’m able to skim read like a pro. Oh yes.

Oh, but a service like filttr would be perfect for me, I hear you cry. A secret, intelligent algorithm that figures out what I want to read so all I have to do in the morning is sit back and trust the magic pixies. Well, that’s the sticking point for me and a lot of people, I guess — and possibly a reason why sentiment classification will never work as a means to cut down personal information overload. I’m an algorithm designer, I have entire farms of pixies at my beck and call, and can I encapsulate an algorithm that will show me exactly what I want to read? Most likely not. My inherent distrust of generalisations means there will always be one tweet that isn’t shown that I would have liked to have read — or even if there isn’t, I’ll think there is, and not want to filter.

Instead, I climb the laborious yet enlightening slope of trying to figure out a system that will more or less get me the information I want without any of that pesky “you might have missed something interesting” feeling nagging at me. I use TweetDeck, as many others do, to get a great visualisation of @replies and DMs as well as the follow firehose. Yet my follows are split into logical (ish) groups, and I can make a couple of friend groups that will streamline the chatter quite effectively. The only problem is keeping said groups up to date — I’m constantly adding new people that seem interesting, and if every follow results in altering two TweetDeck installations’ group settings, urgh. Yet if all new follows end up in the firehose, I haven’t really solved the problem at all.

The other approach which I think will be useful is twofold. Links and conversations. As someone (I forget who) recently noted, as you follow more people on Twitter, you end up being privy to more and more @conversations (assuming you don’t already have ’show all @replies to anyone’ on, which is horrendous to stay on top of and mostly pointless as well). Sometimes it’s useful to join in on these conversations, but at other times, you’re 12 hours late to the party and they’re just not that important. I’ve seen mention of people working on Twitter conversation threading, so I guess this belongs there, but a UI that collapsed conversations (perhaps older than a set time) would save a lot of screen and brain real estate.

Links is the other half of this. It’s quite fun in the morning, while catching up on Twitter, to click on interesting looking links and open them all in tabs then browse through them. Two problems with this. Sometimes people RT the same link and most of the time I forget the link’s origin, so if I want to RT it I can’t give credit. Would pulling out all the links from last night’s twitterstream be useful? I think so, especially if there were some way to associate opened links with the original tweet. Trying to scroll back and see which tinyurl or bit.ly post is actually pointing at the informative Guardian article I have open is a nightmare, but if I had a clean display that pulled out all the links from friends, gave them extra props if linked multiple times, and gathered all relevant comments (possibly including non-friend @replies) while also longurling them for easy back-reference — then hid all the pure-link tweets from last night, I suppose — then URL management might be a little easier.

Of course, a different approach is to manage who I follow. The big question from a NLP/AI point of view here is “can a classifier learn the set {worth following, oh my god no way}?” i.e. if I show an algorithm a twitter account, will it make the same following decision I do?

It seems fairly simple at first pass to jot down a feature set that’s computationally feasible and probably relevant to how I make following decisions:

• userpic present? (more subjective: userpic interesting?)
• twitterer location (Edinburgh, Cambridge or London = likely to be a yes. Rest of world = no preference)
• twitterer language (only follow English tweets, sorry)
• number followers and following (don’t follow followbots; generally follow people with a healthy ratio or people with lots of following i.e. net-celebs)
• volume of updates (don’t follow the super spammy; unlikely to follow people with 1 or 2 updates especially if they plug their product)
• quality of updates (decent number of @replies – person is active participant; lots of URLs to same site – account is a blog bot, unlikely to follow unless I like the blog – ah the irony)
• subject matter, both from bio and from content (if person’s bio matches my interests, likely to be a yes; if person’s recent tweets overlap with areas I’m interested in, similar. maybe they’re using the same hashtag as me, i.e. at the same event.)
• interestingness factor (hardest part to quantify. do they post funny photos? do a lot of people I follow follow them? are they witty? will I benefit from hearing what they had for breakfast?)

Of course, I don’t run through all these factors every time I see a new profile. Unfollowing someone has a low penalty (they might stop following you, oh no!) so it’s very easy to just go “no” at the obvious bots/promoters and “yes” to anyone who seems remotely human and Twitter-savvy, then unfollow them later. But I reckon it would be pretty easy to train up a classifier to decide whether people were worth following, and (a la MrTweet) use shared interests (hello NLP) and friend/follow networks to recommend new people. In fact one of my Google Apps sandbox projects is working on this but it’s not a commercial venture, so I feel quite happy to ramble about it in the hope people will tell me I’m dead wrong and how to fix it.

Another thought which occurs to me when I think about term extraction, frequency and classifying Twitters into subject buckets. Given the total sum of knowledge available to a casual observer (my update history, my network and extended network, my network’s update history) can I use simple clustering techniques to segregate my follow cloud into distinct groups for easier update browsing? I think with some gentle nudging, I could do so for my own network, but establishing an algorithm that could do so for anyone might be more difficult. Perhaps it needs a little Facebook or Friendfeed integration to pull out some more information (I have some seemingly isolated Twitterfriends from university who I guess it’d be hard to cluster at first pass). The question from a commercial point of view is, of course, would anyone use it and would they pay? Generally, as we saw with the magic pixies above, people trust their own judgement better than a computer’s. And yet I think it’d be quite fun to play around with clustering, visualisations and the magic of the Tweetcloud. Especially if we could change the parameters at will to cluster people by similarity (these guys all post loads of links, these all have conversations with each other) as well as overlap in topic/geography/network. Is anyone doing this? Someone must be, surely.

If not, I will.

[Aside: No image, since for some godforsaken reason T-Mobile's mobile broadband service blocks Flickr and then asks for a credit card number over an unsecured connection.]

[Update: Image from dawn_perry]