Thanks so much for this post. You wrote:

“So even if there were a brilliant, perfect, 100% reliable sentiment detection system, it would be wrong 15% of the time, and so humans would want to check every message just in case. ”

But our tests found out that it is worse than what you describe:

http://commetrics.com/articles/fails-validity-test/

Regards
Urs
@ComMetrics

You are correct, no matter what tools are out there, we cannot forget the human element. It pays to listen, but such returns come only after you have established a relationship. Humand forge relationships, not statistics or automated features or a faceless brand/organization.

Looking forward to reviewing your live notes.

Lauren Vargas
Community Manager at Radian6
@VargasL

Social evolution is, and will increasingly be, a slower process than technological innovation. Tools and platforms will come and go but the way people relate to them will remain relatively static.

Conversations once reserved for water-coolers and barber shop gossip now take place online and at the scale this allows, but the underlying need and governing principles are pretty much the same:
Be a mensch,
Don’t kvetch and moan too often,
Do unto others as you would have them do unto you,
Wear sunscreen and never forget your towel.

: )
@pop_art

As a paid SMM service provider, we learned about lesson 3’s wisdom quite some time ago, and have always offered human review with all our tiers of service. The idea of tools only being able to do so much and the need for people to take over really resonates with us. We listen to a lot of the feedback from our clients and the crowds, and when we hear the ‘automagic’ speak overshadowing the tremendous work vendors are doing in this space, one wonders whether its become more of a problem with overselling the tools.

With regard to your comment on tailoring for the tool, I come to this argument with the bias that the paid SMM vendors need to spend more time understanding the requirements and strategies of each opportunity that converts to a client assignment. The other bit is that vendors need to assume some responsibility when it comes to explaining any/all limitations clearly. The basis for this argument is that clients are coming to paid vendors with the expectation that we are ahead of the SMM curve, and vendors need to live up to that role in so far as equipping the client with a tool that fits their specific and unique monitoring/listening requirements.

On your points in lesson 4, I’m just not hearing enough “involve” and “evolve” happening in terms of ridding the problems of “spam” and “malware” from monitoring dashboards and subscribers inboxes. It kind of reminds me of the way MS Outlook kept going through different iterations without tackling the spam problem. The trend most vendors seem to be taking is to continue to inundate machines with the tasks of analysis and “smart” filtering, and giving Google and Microsoft – two companies with no track record on “personalized” service – full license to enter the SMM fray certainly isn’t going to change this trend.

IMHO, the precision and qualitative analysis portion of SMM tools is what needs to be happening around the corner. What’s kept the tools and this space going for this long is the “service” element and that still needs fine-tuning. It’s where the reputation of this space rests and to some extent its worked in keeping the big players with no service experience on their resume at bay for this long.

Joseph
@RepuTrack

For me, automated sentiment analysis in general is compromised but by no means useless. As you say, the focus on individual user needs is key – and while any margin of error is a problem when used as a tool to aid relationship management, it’s much less of a problem when used as a large scale research tool.
Investment banks are happy to use data of uncertain reliability for eye-wateringly large trades via high frequency trading software after all. Understanding and modelling margin of error is the key to usability.

For me the main problem with sentiment automation that is built into SMM tools isn’t accuracy, it’s granularity (I’ve worked in environments where humans code sentiment, and results achieved in that way have their own set of problems).

Essentially, sentiment at document level is pretty hopeless. Sentiment in commercial SMM tools is no better than an interesting toy to us – we’ve experimented a lot with them, but just can’t find a way to make them useful.

And the open source approach is right on the money. there is much more potential in building out custom applications using open source data mining / information extraction tech, than in waiting for licensed SMM tools to meet research needs.

I enjoyed your well written article. In particular, I agree with your point that technology cannot be 100% accurate.

“So even if there were a brilliant, perfect, 100% reliable sentiment detection system, it would be wrong 15% of the time, and so humans would want to check every message just in case. And if all you want is a ‘temperature’ type analysis, well, free tools already do that, and even allowing for error they’re just about good enough.”

For effective social media monitoring, someone still needs to look at the data to weed out spam, sarcasm, decipher acronyms and evolving languages. Thats why my company Brandtology weaves the human intelligence component into our processes.

Will be following your updates. :)

Ashley
Social Media Consultant
http://www.brandtology.com

I also think it’s worth mentioning that no amount of social media activity will help you if you have a crap product or service.

social media can help you find out where you suck and then communicate to your customers how and when you are going to stop sucking.

social media can also tell you and your prospective customers where you shine !

the point is that pretty soon *all* your prospective customers are going to be using social media to find out whether, on balance you suck or shine *before* becoming actual customers.

most of us suck at something, but using social media to communicate effectively how and when you’re going improve can actually be good for your reputation.