06
Jul
10

Why Sentiment Analysis Can’t Work And Why It’s A Damned Good Thing!

Dominique Lahaix, CEO of eCairn

Dominique Lahaix, CEO of eCairn

This is a guest blog post from Dominique Lahaix, CEO of eCairn. eCairn Inc. (@eCairn) is a privately held software technology company, founded in October 2006 that specializes in community and influencers marketing. Headquartered in Los Gatos, California, Dominique (@Dominiq) and other company founders have a cumulative experience of 50+ years in e-marketing, knowledge engineering, collaborative filtering, linguistics, software development and software engineering at well-known companies such as Hewlett-Packard, Sun, Xerox, eBay and Motorola.

Let’s start with an example.

Imagine you work for Dell and you have to rate the following conversations to be examples of positive, neutral or negative sentiment:

  • HP is good.
  • Dell is good, but I still prefer HP.
  • The new Dell PC is good but the previous one worked better.
  • The new Dell PC is good but I miss the look of the previous one.
  • HP works great but I hate them.
  • Dell works great but I hate them.
  • Dell is as good as HP.
  • HP and Dell are the same crap, maybe Dell a little bit less.
  • HP and Dell are nice entry-level products.
  • Dell is good if you can afford it.
  • Dell is good but exclusive.
  • I would only recommend Dell’s PC to small businesses.
  • Dell is the Dom Perignon of netbooks.
  • Apple is to Dell what Saint Amour is to Beaujolais Nouveau.
  • I worked too much on my Dell last night and I got sick looking at the screen.
  • Dell is only good for gaming.
  • HP is like it was in the Packard’s time.
  • HP is like what it was in Carly’s time.
  • No wonder why the Dell stock is going south.
  • I love HP (from HP’s PR agency or Director).
  • I hate HP (from an employee recently fired).
  • A tweet repurposing the one above without any additional comment.

Hard to rate isn’t it? These are fairly standard sentences, not corner cases. There is no irony and no borderline use of language, but it just shows that:

Bookmark and Share

  • Sentiment is subjective.
  • Sentiment is role based: depending on whether you’re the brand manager for Dell overall, or in charge of the new Dell notebook, or the Small Biz vertical, QA manager, Investor relations, or looking into competitive analysis, you may have different views on what positive means.
  • Language to express sentiment is context dependant and cultural (wine analogy).
  • Language to interpret sentiment is context dependant and cultural (see the HP history analogy).
  • Sentiment analysis has to factor in the “who,” and account for real and pseudo duplicates. Same conversation from different persons, similar conversations but from the same person. Everybody is repurposing nowadays, re tweeting, using ping.fm or alike, not counting the huge amount of “robot “sites or blogs that just sit there to attract traffic and ads.
  • “Social Media” is not a valid sample of any user base, even when the user based is defined as internet users. As an example: the open-source/Linux community is way more vocal than the windows one. Just taking data from the river of news from community a statistical heresy, except if you want to study Linux fans.

That’s why solutions claiming to do automatic sentiment ends up with 60+ neutral, 70% precision and manual options to override the machine produced sentiment.

It just does not make sense. Putting everything neutral may well be a better bet from a recall and precision standpoint.

I’m not saying it can’t be done. It just can’t be done “generically.” Now if you build a solution for sentiment analysis specializing in the stock exchange community, this is another story. You can build up dictionaries, invest in a learning algorithm, train it… and yes, that sounds doable but it would be very expensive to set up and the applicability of this would be limited to “stock exchange,” even maybe to stock exchange in 2010.

So forget this Holy Grail, stop wasting $$$ on low quality results and come back to the initial objectives of the brand:

Get a sentiment on its brand from a specific audience with a reasonable investment.

There are alternatives to reach this goal and the good news is that math comes to the rescue!

  1. Why not sampling? Marketers have always used focus group and samples, why not extending that on the social web?
  2. Why not rating manually? When you zoom in a specific community, you can start doing that. We looked at the top Mommy bloggers that we’ve mapped (top 3,500) and went down to 2000 discussions about Pampers in the last six months. One can do a good job rating three conversations per minute, so it’s roughly a 12 hour job, at $20 per hour, that’s $240 over six months.
  3. Moreover, as there is more consistency between the conversations that you rate, the quality of the rating would be higher. It’s easier as an example to establish a standard to rate blog conversations from Mommies on Pampers, than to build one for any conversation on Pampers (that would go for analyst reports on Procter & Gamble to sustainability conversations from experts!). Conversations are more consistent, more alike and easier to rate with a focus. You also get specific results that are specific to your key target communities and way more actionable results.
  4. While you rate, you also spot insights, key conversations to share, ideas for content marketing and more.
  5. Also, doing it this way, you will actually find your promoters and detractors and build specific outreach plans.
  6. And as you connect conversations to people, you will see who’s moving in positive territory and can link that to your specific action plans, see whether cluster of influencers are moving in the right direction.

In addition, I’m still wondering what type of actionable plan a brand can take when its “sentiment” drops by 3% when accuracy is claimed to be 70% maximum… The Motrin case went wild over the weekend, Domino Pizza within hours. So for crisis management, investing in Proactive ORM and building up a solid base of fans within the target community is a much better option (Ford’s approach).

And last but not least, when 93% of consumers say they want brands to engage in Social Media, I doubt that they mean engaging with algorithms and I bet they are expecting real and empowered persons. But that’s another story.

Bookmark and Share


10 Responses to “Why Sentiment Analysis Can’t Work And Why It’s A Damned Good Thing!”


  1. July 29, 2010 at 4:40 PM

    nice post. thanks.

  2. July 20, 2010 at 8:51 AM

    Very interesting. Seems like the problem is to be able to create the algorithms – and the process of training them – for very specific corpora. Client-specific, vertical specific.

    Dragon Naturally speaking requires at least a couple of hours to train to an individual.

    Certainly a sentiment analysis engine worth having would also require some degree of training to learn about what is and is not accurate given a specific collection of documents / specific references to specific entities.

  3. July 19, 2010 at 10:31 PM

    Very interesting thoughts on this one.

    First off, I’d like to point out that it is true that automated sentiment analysis does have its drawbacks, but then I’d like to point out that there are different providers that can provide human sentiment analysis that could better interpret the sentiments of statements and conversations. As a company that does this kind of work, we have seen the discrepancies that can arise when relying on a computer to interpret human emotion. This just means that we cannot assign a sentiment to a value and assume that the statement will always be true (i.e. wonderful = positive, whenever used in a sentence), hence human analysis.

    Now on your other point, sampling is a good idea, if you already have a current market to assess. But in the long run, it would be inefficient to focus on just a specific market especially with regards to brands with a broad scope. Likewise, a company may not even realize that they are attracting a certain market even more than their target market. Thus by limiting their view, they restrict themselves to seeing only what they believe they need to see (or know) rather than what is really out there. Sure, common sense would dictate that Pampers would be talked about more by mothers, but what then of products such as, say, Mcdonalds? A brand like this may target younger generations but then by sentiment analysis, they find out that grandmothers/grandfathers find their happy meal toys inadequate and have been discussing it online. Sure, sampling has its uses, but it definitely cannot apply to all.

  4. July 18, 2010 at 4:44 PM

    It’s posts like this that keep me coming back and checking this site regularly, thanks for the info!

  5. July 9, 2010 at 8:01 AM

    Hi Pascal,

    Thanks for the comment. I’m actually making two points:

    1. The technology is not ready to provide a generic algorithm that rank with enough quality individual conversations as negative, neutral or positive (BTW, why 3 values, isn’t a 4 value system a better way to force choices ?).
    2. There is a huge investment required in:

      • - Cleaning the corpus (a large and structured set of text, now usually electronically stored and processed), normalizing it (the social web is a huge mess, conversations on Twitter, blogs, Facebook and Wikipedia or have nothing in common; different size and different styles)
      • - Machine learning
      • - Training and dictionary-building for getting high-quality results

      As a result, this makes sentiment analysis non-economically viable for most projects.

    3. Twitter, Facebook, Wikipedia and blogs are not a giant call center and Social Marketing is not Customer Support.
    4. Brands can’t get “social value” out of it – learnings, trust, relationship building – if they either use machines to read and rank the conversations or even if they outsource to a workforce that’s measured towards minimum time spent and lower interaction cost as possible.

      I agree with you that manual rating is difficult, too, and to do a proper job, the person in charge of the rating should have an understanding of the context of the rating and be able to dive in the “who” and in the “why”s.

    As to your example, “reducing the mention/ conversation collection”, what makes you think that brand should focus on positive and negative mentions – or even that they should focus on conversations mentioning them?

    I would rather recommend focusing on who matters (i.e. people and communities that are in the target) and listen (not monitor) what is said. When a Forrester analyst mentions most of your competitors – and not yourself – in an article that is positive, I can tell you this is a critical conversation ;-).

    Best,
    Dominique Lahaix
    CEO, eCairn
    conversation@ecairn.com

  6. 6 infoglouton
    July 6, 2010 at 2:06 PM

    Interesting article, but you don’t tell why sentiment analysis *can’t* work (you just say that it *does not* work), nor why it is a good thing (you only say that there is a manual alternative. I suppose you meant to say that it’s a damned good thing for PR businesses who bill their time). Also, the examples about HP and Dell are equally difficult for manual ratings, it does not support the fact that automated sentiment analysis cannot reach the accuracy of humans.

    Specifically 1, 3, 4, 5 and 6 can all be done in combination to automated sentiment analysis. Only 2), manual sentiment analysis, cannot be done using automated sentiment analysis for obvious reasons.

    So the only remaining case against automated sentiment analysis seems to be the accuracy. I don’t know if 70% is the state-of-the-art accuracy of sentiment analysis systems, this figure seems to be a bit low. Nevertheless, a 70% accuracy is enough to draw many statistically significant conclusions given sufficiently large samples.

    There are other usses to automated sentiment analysis, it’s not just about conducting surveys on the brand perception. Automated sentiment analysis could be used to reduce your mention/conversation collection to a much smaller set. By adjusting the parameters of your sentiment classifier, you can keep only positive and negative mentions for which there is a high level of confidence (putting the rest in the neutral bag) and focus on these mentions. The precision could then reach over 95% (at the cost of recall obviously) and this strategy could help prioritizing mentions that require your attention (interacting with the customer, making changes to your sales process, etc.).

    Also, conducting a survey over the course of 6 months is not an efficient way to quickly deal with negative mentions that could spread virally online. Sentiment analysis provides a huge opportunity to harness the power of real-time monitoring.

    Pascal

    • July 9, 2010 at 9:41 AM

      Dominique Lahaix did a good job of explaining why sentiment analysis does not work, and it doesn’t. The fact that it does not work is not really a blessing for PR agencies who bill their time: if it worked, there would be many other ways to create value and bill time. On a lighter note… if automated sentiment analysis can be made to work 95%, maybe there should be one such app in every family, as an instrument for husband and wife (and parents and children) to communicate with each other. THis would bring to families at least a 50% increase in how everyone understands what the other means…:)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Subscribe & Share

Subscribe By RSS  Subscribe By Email

Bookmark and Share

Like on Facebook

Like Jennifer on Facebook

Follow On LinkedIn

Jennifer Lindsay Digital on LinkedIn

Latest Tweets @jennifered

The A-List Show

Jennifer Lindsay, Featured Podcaster on BlogTalkRadio

Questions for all guests are crowd-sourced. Tweet, Facebook or email in your questions!

Upcoming Speaking Engagements

Instructor: Facebook Marketing: Build a Fan Base and grow your business with a Facebook Page. Online course- mediabistro. November 10 - December 15, Wednesdays, 9-10 PM EST. Click on the link for additional details and to register.

Speaker: San Francisco Blogging Club Invited guest speaker. Monday, November 15, 5:30 PST, Minna Gallery, San Francisco.

Speaking RequestsInterested in having me speak at your event? Contact me [at] jenniferlindsay.com

Rock the Red Pump

The Red Pump Project

Top Rated

My Commitment

Blog With Integrity

Top Clicks

  • None

Archives


Follow

Get every new post delivered to your Inbox.

Join 113 other followers

%d bloggers like this: