Can Social Media Measure Customer Satisfaction?

Elliot Bricker, Director, Product Management, NetBase MA RC H 2 01 1

Reviewed by an independent team from the Walter A. Haas School of Business at the University of California at Berkeley. The team was led by Vito A. Sciaraffia PhD candidate in business economics and strategy.

Executive Summary
Because customer satisfaction is a key predictor of financial performance, businesses have invested a lot of money and resources in tracking the satisfaction of customers through surveys and benchmarking. Social media moved into the mainstream in 2010; savvy executives soon recognized that the millions of blog entries, micro-blogs, status updates, and comments that consumers post every day could become a new source of customer satisfaction data one that is significantly faster and less expensive than traditional survey methods. The question is, does sentiment expressed in social mediathat is, whether online posts are positive, negative, or neutralcorrelate with established customer satisfaction metrics? To answer that question, NetBase compared the Net Sentiment Score in the NetBase Insight Scorecard to published scores from the American Consumer Satisfaction Index (ACSI) and found a high correlation (Pearson Product Moment Coefficient r=.773). In this white paper, we discuss our analysis and also share research findings that will help you benchmark your online sentiment scores with industry peers. We also discuss the roles that Passion Intensity and Share of Buzz play as social metrics that measure distinct facets of the customer experience.

Can Social Media Measure Customer Satisfaction?

Table of Contents
Customer Satisfaction Matters at the Bottom Line .....................................................................................4 Social Media: A New Source of Customer Satisfaction Data ......................................................................4 Does Online Sentiment Correlate With Customer Satisfaction? ...............................................................4 NetBase Insight Scorecards Track Three Key Metrics of Online Brand Equity................................... 5 Measuring Sentiment and Passion With the Science of Language .......................................................... 5 NetBases Net Sentiment Score ............................................................................................................................... 7 Correlating Net Sentiment to ACSI ........................................................................................................................ 8 NetBases Passion Intensity Score ........................................................................................................................ 10 Is Passion Intensity Really Different From Net Sentiment and Buzz? .................................................... 11 Bringing it All Together: The Brand Passion Index........................................................................................ 12 Conclusion....................................................................................................................................................................... 13 Methodology .................................................................................................................................................................. 13 Appendix ......................................................................................................................................................................... 14 Related Reading ........................................................................................................................................................... 15 About the Author ......................................................................................................................................................... 15

Can Social Media Measure Customer Satisfaction?

Customer Satisfaction Matters at the Bottom Line

Businesses know that customer satisfaction is a predictor of financial performance. Satisfied customers generate more revenue. They are also more profitable because it costs much less to market to existing customers than to acquire new ones. As a result, businesses have invested a lot of money in tracking the satisfaction of their customers through a variety of customer satisfaction (CSAT) and voice-of-customer (VoC) surveys. Email surveys are the most common data source, followed by telephone incident follow-up surveys, and finally self-service follow-up using VoC tools that intercept users during visits to a web site. There are a variety of established measurement methodologies such as the Net Promoter Score (NPS) or SERPVAL/SERVQUAL and benchmarking services such as the American Customer Satisfaction Index that serve to make CSAT survey data more meaningful and actionable. However, all survey-based tracking methods have a common set of shortcomings: They rely on limited samples of customers Data collection takes time, slowing down responses to potential customer satisfaction issues Because of the costs involved, most companies can only afford to track their own brands

Social Media: A New Source of Customer Satisfaction Data

Forward-looking businesses have recognized that a new source of customer satisfaction has emerged: social media. Every hour, consumers make more than 500,000 new blog and micro-blog entries, status updates, and comments in social media. They are raving about companies and brands that they like and spreading the word about the bad experiences they have had. In fact, social media is where new impressions propagate first and fastest today.

Does Online Sentiment Correlate With Customer Satisfaction?

Social media offers a fast, inexpensive way to measure customer satisfaction for both your brand and your competitors brands. As executives evaluate it as a strategic data source, many are asking the key question: How can social media help us to measure customer satisfaction? In theory, an aggregate measure of online sentimentthat is, whether online posts are positive, negative, or neutralshould correlate with established customer satisfaction metrics. In this white paper, we will explore that question. We will look at how the Net Sentiment Score in the NetBase Insight Scorecard compares to scores calculated with popular CSAT

Can Social Media Measure Customer Satisfaction?

methodologies. Our findings demonstrate a strong correlation between the American Consumer Satisfaction Index (ACSI) and NetBase Net Sentiment Scores derived from social media opinions. We will also share research findings that will help you benchmark your CSAT scores with industry peers. In addition, we will look at the roles that Passion Intensity and Share of Buzz play in understanding customer satisfaction. Finally, we will discuss the role that social media can play in helping executives to understand the underlying root causes of customer satisfaction issues, so that they can make smarter business decisions faster.

NetBase Insight Scorecards Track Three Key Metrics of Online Brand Equity
NetBase Insight Scorecards were designed to give businesses a reliable way to measure, enhance, and protect brand equity in social media. They deliver up-to-date metrics on three key aspects of your customers experience: Share of Buzz - How much people are talking about your brand Net Sentiment - How positively they perceive your brand Passion Intensity - How emotionally charged their feelings are

These three metrics, along with measurements of key conversation drivers that you specify, are displayed with 12 month historical trending and benchmarked against your key competitors. Because they do not require expensive consulting fees or time-consuming data collection and assembly, NetBase Insight Scorecards give you a fast way to stay on top of your consumers experiences with your brand.

Measuring Sentiment and Passion With the Science of Language

ConsumerBase, the data source accessed via Scorecard, is a social intelligence warehouse containing a full years worth of social media commentary across more than 95 million

Can Social Media Measure Customer Satisfaction?

sources, from forums and blogs to Facebook and Twitter. We use the most advanced natural language processing (NLP) engine to read, understand, and categorize every posting in ConsumerBase according to the sentiments, emotions, and key ideas that your consumers have expressed. Then, we aggregate these sentiments, emotions, and ideas into the metrics that you see in your Scorecard: Share of Buzz, Net Sentiment, and Passion Intensity. NetBases NLP engine represents a big leap in accurately analyzing content from the social media universe. Unlike tools that infer sentiment based on statistical keyword matching, NetBase understands sentence grammar at a deep level and delivers over 80 percent accuracy. This approach does not involve counting words or analyzing text; NetBase reads sentences, evaluates grammatical sentence patterns, and organizes results to be fully searchable on a wide variety of attributes. We analyze social media content in two steps: parsing and normalization. First, the NLP engine parses each sentence it captures from social media at a very deep level. This process is similar to the sentence diagramming that students do in a high school English classit identifies and links the subjects, objects, verbs, adjectives, and other linguistic patterns in the sentence to extract deep and accurate understanding of what is being said. By analyzing this connective tissue within each sentence, our NLP engine can account for the complexities in language that make keywordmatching algorithms inaccurate. Anaphora resolution is a key part of the parsing process that ensures that sentiment-rich associations are not missed. Anaphora are grammatical substitutes that refer back to your brand name in sentences that may otherwise be missed in most social media analysis tools. Typically, a pronoun (e.g. it, her, him, their, they), anaphora refer back to another unit, as the use of "they're" and "they", and "them" refer to M&Ms in the sentences: M&Ms are sweet and delicious. I remember always hearing that they[M&Ms] melt in your mouth and not in your hands. We're going to buy more[M&Ms] today down at the store because we love them[M&Ms]. So we see that the anaphors obey what linguistics call binding conditions. Anaphora resolution requires semantic understanding that is even today undergoing active academic and business research in the field of natural language processing.

Can Social Media Measure Customer Satisfaction?

Next, our NLP engine normalizes all the parsed sentences to make them easy to aggregate into the metrics shown in our Scorecards. It takes sentences (what we call sound bites) and stores them, based on the type of insight they reveal, in a single, consistent format, regardless of the structure of the underlying sentences. Normalization is a fundamental part of NetBases unique value because it allows our solutions to expose not just positives and negatives but also deeper-level insights such as passion. Working with many of the most sophisticated brands in the world, we have optimized the NetBase NLP engine specifically for understanding social media and the Web. In addition to standard English, our extensive lexicon includes a wide variety of urban words and phrases, alternative spellings, and abbreviations common in social media, as well as common misspellings. We are constantly incorporating new rules into this lexicon based on the work of our internal linguistics experts, ongoing testing that we do using crowdsourced human evaluators, and feedback from customers.

NetBases Net Sentiment Score

Automated sentiment analysis focuses on analyzing the content of online posts, determining whether they are positive, negative, or neutral, and aggregating the sentiments detected into a single generic score. The Net Sentiment Score computes a ratio of positive and negative mentions of a topic. Our analysis is localized to individual sentences, our sound bites; we do not view an entire post as a single sentiment. The formula for Net Sentiment is:

A Net Sentiment of 100 means that all mentions of the topic are positive, and -100 means all mentions of the topic are negative. The average Net Sentiment Score across the thousands of brand names, companies, people, and other product names mentioned in ConsumerBase is 32 This indicates that overall, the 32. chatter about brands and experiences is somewhat positive. The NetBase Insight Scorecard charts the Net Sentiment Score in a time series for the brands that you track. This view makes sentiment data more actionable because you can easily see its directionality. You can also see a count of positive sound bites (green area in chart on the next page) and negative sound bites (red area). The y-axis secondary scale (far right) displays a scale for the volume of sentiment polar sound bites.

Can Social Media Measure Customer Satisfaction?

Below are some examples of a few industry- or category-specific Net Sentiment Scores. They have been computed over the course of a single year and represent a comprehensive selection of companies, their related products, and brands for each industry.

Additional analysis was conducted to prove that the distribution of Net Sentiment Scores does not cluster for a specific industry. In other words, Net Sentiment Scores follow a typical bell curve distribution where values are distributed fairly evenly around their mean and tail off at the extremes of high and low scores. Please see the Appendix for more details.

Correlating Net Sentiment to ACSI

In order to look at how Net Sentiment compares to established measures of customer satisfaction, we computed Net Sentiment Scores for 12 retail businesses (retail, wholesale, or department stores) and compared them against the academic methodology of the ACSI, developed at the University of Michigan. These scores are the aggregate of 12 months of data that span February 1, 2010 until mid-February, 2011. The ACSI releases industry results monthly, and the scores reflect the most recent installment February 2011.
The American Customer Satisfaction Index (ACSI) is a uniform, national, cross-industry measure of satisfaction. The distinguishing feature of the ACSI methodology is its patented cause-and-effect approach to customer satisfaction measurement. Three questions comprise the assessment of satisfaction: What is your overall satisfaction with our product or service? To what extent has our product or service met your expectations? How well did our product or service compare with the ideal? Organizations normalize (weight) and average the three ratings to produce a score from 0 to 100.

Can Social Media Measure Customer Satisfaction?

The comparison between the NetBase Net Sentiment Score and the scores reported by the ACSI organization for the same time period showed a high (r=.773) Pearson Product Moment Coefficient meaning there is a good correlation between the two scores.

Store
Net Sentiment Score Target Kohl's J.C. Penney Sam's Club Lowe's Walgreens Best Buy Macy's Home Depot Rite Aid CVS Wal-Mart 78 81 80 78 77 77 77 76 75 75 74 73 51.3 70.1 65.3 75.0 59.4 52.5 66.6 66.2 58.0 51.8 50.5 40.6

To assure that the correlations of Net Sentiment and other customer satisfaction methodologies didnt reflect the state of one industry, we also looked at a cross-industry spectrum of profiles. This included companies from the automotive, airline, financial, Internet retail, Internet travel, CPG, and grocery store verticals.1 The correlation approached a similar coefficient to that measured for the single retail store industry. As can be seen in the chart on the next page, that number was .714. This again implies that NetBases Net Sentiment Score serves as a strong indicator of CSAT.

The companies included: Bank of America, BMW, Charles Schwab, Delta Airlines, eBay,E*TRADE, Expedia, Hershey, JP Morgan Chase, Kellogg's, Kia, Kraft Foods, Kroger, Lincoln Mercury (Ford), Nestl, Netflix, Newegg Orbitz, Publix, Safeway, Southwest Airlines, TD Ameritrade, US Airways, Wells Fargo, and Whole Foods.

Can Social Media Measure Customer Satisfaction?

NetBases Passion Intensity Score

Marketers know that passion intensity matters. It drives word-of-mouth both good and bad. It generates brand loyalty and repurchase. But until now, companies did not have a way to quantify how strongly consumers feel about particular brands. The NetBase Insight Scorecards include a Passion Intensity Score a number that ranges Score, from 0 to 100. This score complements the Net Sentiment Score by adding another dimension of the customer experience: emotion. The formula is:

To calculate the number of mentions in each category, the NetBase NLP engine identifies comments and posts that use a specific set of emotions and qualities. Measurements of the "Love" emotion aggregate linguistic associations to feelings such as love, adore, fan, luv, thrilled and many more. Additionally, the "Love" emotion accounts for positive subjective qualities such as brand associations to terms like incredible, fantastic, tremendous, amazing, amazing, and so on. Measurements of the "Hate" emotion cover expressions such as hate, despise, loathe, detest, and disgust, with the added complement of negative subjective qualities that include detest, disgust, shocking, stink, horrific, appalling, shocking, horrible, awful, terrible, suck, stink, and so on. The denominator of the formula combines the total number of these emotion-laden love and hate mentions with a measure of other non-emotional but subjective qualities such as awesome. best, worst, suck, rocks, good, difficult, or awesome

Can Social Media Measure Customer Satisfaction?

10

As with Net Sentiment, we calculated an average Passion Intensity Score of 30 across thousands of brands, products, people names, and issues. To give you an idea of the range of passion intensity, here are some measurements that show scores at both ends of the spectrum: Passion Lower Passion Intensity J.P. Morgan Chase 8.4 iShares 6.2 Nexium 14 Higher Passion Intensity Windows XP 62 Volkswagen 63 Grand Theft Auto 70 Marmite 81

For the 12 stores we looked at in the ACSI comparison, we calculated the Passion Intensity Scores seen below. The scores aggregate over 12 months of analysis for individual mentions expressing emotions and subjective-quality insights:

Store
Target Kohl's J.C. Penney Sam's Club Lowe's Walgreens Best Buy Macy's Home Depot Rite Aid CVS Wal-Mart

Passion Intensity
26.2 41.9 15.0 41.3 31.8 37.9 90.0 43.6 30.9 46.0 34.4 35.6

Is Passion Intensity Really Different From Net Sentiment and Buzz?

NetBase believes that there is no single score that can capture all facets of online brand equity. Thats why our Scorecards contain three metricsShare of Buzz, Net Sentiment, and Passion Intensitywith in-line comparisons to competitors and historical values. Our research has shown that each of the metrics we calculate and display differs greatly from the metric of sentiment. For example, the Passion intensity metric is very low in correlation (r=0.10048 Pearson Product Moment Coefficient) to the Net Sentiment Score.

Can Social Media Measure Customer Satisfaction?

11

Affective discussion in social media is not always an associative measure of sentiment. For example, where sentiment can be very positive, ardor and fervency can be said to be rather blas.

The low levels of correlation between our Net Sentiment and Passion Intensity Scores attests to the fact that each metric stands on its own measuring an aspect of your brands consumer satisfaction, with its own distinctive implication in your analysis. Similarly, Buzz Volumethe number of occurrences of sound bites for the topic of interestis a facet of the customer experience that is distinct from Net Sentiment. Consumers may be very chatty or relatively silent about a brand, but the data still proves that no correlation exists in the volume of chatter to their overall expressions of polarity in sentiment. A correlation analysis showed that the direction and degree (closeness) of linear relations between buzz and sentiment was quite low at 0.01159.

Bringing it All Together: The Brand Passion Index

In order to have one crisp visualization of the three metrics of Buzz, Passion Intensity, and Sentiment, NetBase provides a Brand Passion Index. The Brand Passion Index is an intuitive visualization that lets you quickly analyze consumer sentiment, buzz, and passion intensity across multiple brands and view historical change in a single chart. The map has four quadrants: Like, Dislike, Hate, and Love. You and your competitors are placed on one of these quadrants based on the

Can Social Media Measure Customer Satisfaction?

12

Passion Intensity that consumers express (Like versus Love) and their Net Sentiments (Like versus Dislike). The size of the competitors bubble indicates the volume of posts for that competitor.

Conclusion
Social media is the next strategic source of consumer insights and of competitive advantage. In this white paper, we have shown that consumer satisfaction expressed in social media is both important and measurable. We have looked at how NetBases Net Sentiment Score correlates with established measures of customer satisfaction (CSAT). We have also examined Passion Intensity and Share of Buzz as distinct facets of a consumers experience with a brand. Social media insight and analysis can play an important role in the next wave of customercentric businesses. The businesses that will get ahead of the competition are the ones that are starting down this path today.

Methodology
The data and statistical methodologies have been reviewed by an independent team from the Walter A. Haas School of Business at the University of California at Berkeley. The team was led by Vito A. Sciaraffia, PhD candidate in business economics and strategy. Mr. Sciaraffia holds an MS in business administration from UC Berkeley, an MBA with concentration in finance and statistics, an MA in finance from the University of Chile, and a BS in economics and management from the Pontifical Catholic University of Chile. Additionally, he holds several academic and professional certifications in statistics. If you would like additional information about this white papers supporting data, please contact support@netbase.com.

Can Social Media Measure Customer Satisfaction?

13

Appendix
In evaluating Net Sentiment Scores as indicators of customer satisfaction, we were concerned about clustering in specific industries, where a majority of scores fall close to the mean rather than following a typical bell curve distribution. To test our hypothesis that the distribution of Net Sentiment Scores does not cluster for a specific industry, we took over 125 womens and mens fashion store brands and computed the normal distribution for each of their scores. We found that the variance, which describes the spread of the distribution about the mean value, indicates that there is good scattering. Additionally, there is enough mass (observations) to both sides of the mean. The lower and upper tails of the distribution are as expected (gradually more unlikely to observe low scores and high scores) with values nicely distributed between Net Sentiment scores of 28 to 80. This says that the Net Sentiment Scores for each of the stores is not closely clustered around the mean value for all brands. The 2D Histogram of the mass function show other good distribution characteristics: A mean of 57.99 for Net Sentiment Score and a standard deviation of 13.25 shows that one standard deviation on either side of the mean approaches 68%, two standard deviations - 95%, and three - 99%.

Some representative Net Sentiment Scores for both the low end and high end of the spectrum of the 125 data points are:
Guess Jeans and Accessories Rainbow Shops Tous Champs Sports 28.4 29.0 29.9 31.5 Armani A/X GapMaternity Heritage 1981 Tag Heuer Boutique Boss Hugo Boss 80.0 81.5 81.8 82.2 82.8

Can Social Media Measure Customer Satisfaction?

14

Related Reading
Net Promoter Industry Report 2010, Satmetrix, 2010. Social Media: Are You Listening to the Voice of the Customer? A Joint Research Paper from Verint Systems and TSIA, 2011. How Does NetBase Achieve the Best Accuracy for Understanding Consumers Online? NetBase, September 2010. The Worlds Most Valuable Brands. Whos Most Engaged? Wetpaint and Altimeter Group, July 2009. ACSI Score & Its Calculation, Jeffrey Henning Mar. 11, 2009, <http://blog.vovici.com/blog/bid/18135/ACSI-American-Customer-Satisfaction-Index-ScoreIts-Calculation>.

About the Author

Elliot Bricker participates in new feature design, pre- and post-sale customer support, and other product success areas at NetBase. Elliot began his career at Information Builders working in the areas of business intelligence, data connectivity, expert systems software, and data warehousing. Within numerous start-up companies, Elliot has focused on applying machine learning as well as semantic search and categorization of enterprise content to solve business problems. Most recently, his interests have centered on opinion mining, specifically bringing together social intelligence with business intelligence. He also has expertise in text mining, analytics, data visualization, data warehousing, and optimization algorithms. Elliot has contributed to numerous patent applications, both in the United States and internationally. He holds BS and MS degrees in Computer Science.

Additional Acknowledgments
NetBase would also like to thank Linda Sonne-Harrison of Giant Stride Marketing Group for her helpful review, feedback, and editing in regards to the content of this paper.

Can Social Media Measure Customer Satisfaction?

15

About NetBase
NetBase Social Media Insight & Analysis helps marketing teams make smarter business decisions faster. We deliver tools and Scorecards that give market researchers and brand managers a reliable way to understand online brand equity, analyze and compare consumer passion, and generate deep insights that answer their why questions. Serving hundreds of corporate customers, our products were developed in partnership with five of the top 10 CPG companies, including Coca-Cola and Kraft, and are used by four of the top 10 market research firms, including J. D. Power & Associates. Based in the heart of Silicon Valley, NetBase is a privately held company.

For more information, visit: www.netbase.com

@Net_Base

NetBaseInc

NetBase Solutions, Inc. 2087 Landings Drive Mountain View | CA 94043 P 650.810.2100 F 650.968.4872 www.netbase.com

201 1 NetBase Solutions, Inc.