Pages Navigation Menu

Social Listening and Flawed Sentiment Analysis – Don’t Throw Baby Out With Bathwater

Social Listening and Flawed Sentiment Analysis – Don’t Throw Baby Out With Bathwater

It has been over a month since my last blog post! The ProKarma acquisition of Mantis generated great opportunities for me to share our combined story to a combined client base. I’ve also done several client presentations for topics ranging from social listening and analytics with Pulse Analytics, to corporate Business Intelligence (BI) and Data Governance strategy. I attended SapphireNOW in June, and products like HANA, Design Studio, and Lumira make me want to jump back into the BI world. In-memory computing, and data visualization, have come a LONG way – and the improvements will only enhance the end-user experience. I might write a subsequent post, and include some screenshots, so you can get a glimpse of the “new possible.”

For now, let’s get back to what prompted this blog post title. I was giving a Pulse Analytics demonstration last week, and it got side-tracked with a deep discussion on the flaws of sentiment analysis. To set further context, this gentleman was one of the most intelligent people I’ve met in recent memory. He nailed everything that is currently wrong with Natural Language Processing (NLP) – with slang and context two of the biggest stumbling blocks. He finished his outstanding explanation to the other people in the room, and then he looked at me.

My response: You are absolutely right, sir.

As part of Pulse Analytics ongoing development, we periodically kick the tires on different text analysis solutions. And although they continue to get better, none of them are infallible. Let’s start with slang first. Guess how sentiment analysis would score the following:

Oskar Blues Ten Fidy is the F****** Bomb!

Sentiment analysis would see “bomb”, and it would see the profane adjective, and it would score the sentiment as strongly negative. But we have to evaluate the tone of the author, and the preferred communication style of the social channel, and conclude that this brand mention is actually strongly positive (I concur, Ten Fidy is a great Russian Imperial Stout beer). In contrast, how should sentiment analysis score the following?

My employer dropped the F****** Bomb on me – I got laid off.  Side note: Now replace “I got laid off” with “I got the promotion”.

What’s a poor sentiment analysis engine to do?!? Now, let’s look at a less profane example of context from an actual Pulse Analytics collection:

 Social Listening and Flawed Sentiment Analysis

In this case, there is zero context. The post literally was “sick shot”. By default, the sentiment analysis engine scored the document as strongly negative. However, clicking on the document from within Pulse Analytics takes the user to the original source document. By viewing the picture posted to Instagram, and the context of the other comments, we can conclude that the original author was paying a compliment to the photographer! Let’s stick with the “sick shot” phrase for another example:

Social listening and sentiment analysis

We have more context in this example, and the sentiment analysis engine “correctly” relates “sick shot” to the word “awesome”. But what if we were monitoring for the opposing team – is the score still correct in regards to brand health? Even if a true hockey fan has an appreciation of a “sick shot”, regardless of who scored the goal, we may not want this document to skew the sentiment results of our overall brand health. In this case, we may elect to mark them as Neutral or Exclude them from the sentiment score.

These are just a few examples where sentiment analysis can go awry. But do we throw away a social listening strategy (the baby), or even the sentiment scores for the entire collection, based upon a few exceptions (the bathwater)?

My answer is a resounding no!

Social listening is about so much more than a sentiment score. People are talking about your brand. Do you know…

  • WHO is talking about you?
  • WHAT they are saying?
  • WHERE these conversations are?
  • WHEN they prefer to engage on social?
  • WHY they are talking about your brand?
  • HOW they feel as they are talking about your brand?

Sentiment analysis just covers that last point! There is a lot of actionable intelligence in those other 5 points even if you never considered the sentiment score. And that actionable intelligence includes allowing the brand to find the conversations and engage the authors with the appropriate context to improve their customer experience.

One final note: I still see value in sentiment scoring that is mostly accurate. It gives the brand a headstart on judging overall brand health. Pulse Analytics even allows users to change the sentiment on an individual basis (like my exclusion of the word Johnson in the screenshot above) or en masse. I could search for the string “sick shot” and change the sentiment for all mentions of the phrase to be positive, about 1,200 for this example, with just a few clicks.

So what is your stance on brands needing to adopt a social listening strategy? What are your thoughts regarding sentiment analysis?

Photo Credit: Murner.Nerrenbeschwerung.kind. Licensed under Public domain via Wikimedia Commons.

Related Posts Plugin for WordPress, Blogger...

Brian Vickery

I love my Vickery Girls - and grandsons! My career has blessed me to the point I was able to start a new consultancy in 2018: Analytic Integrity. I look to provide analytic experience, and business integrity, to an Analytics world while helping data-driven organizations mature. I enjoy teaching and coaching, watching football and basketball, and playing tennis. I graduated UT-Austin.

More Posts - Website

Follow Me:
TwitterLinkedInGoogle PlusYouTube