23/10/2010 Social networks offer free access to th…

Social networks offer free access to the

beating pulse of the nation
Forget opinion polls and focus groups – data-mining on sites such
as Twitter offers the best, and cheapest, way of tapping into the
zeitgeist

John Naught on
The Observ er, Sunday 2 4 October 2 01 0

Tw itter allow s us to forecast trends in film s as well as the stock m arket. Photograph: Sarah Lee

One of the few comical aspects of the spending review is the frantic attempts by all
concerned to predict how the victims of Osborne's axe will respond. The major
newspaper groups and the Tory party will of course be deploying the usual – expensive
– steam-age tools: opinion polls and focus groups. The cash-strapped Labour and Liberal
Democrat parties may have to resort to cheaper techniques – inspecting the entrails of
slaughtered goats, perhaps. In the interests of levelling the playing field, therefore, this
column offers them a better idea: intelligent data-mining on Twitter.

It's taken a while for the penny to drop, but finally the world is waking up to the fact that
the phenomenon of social networking might actually tell us useful things about what's
happening out there in the world beyond the Washington Beltway and the Westminster
village. Not only that, but the resulting data might even be useful for predicting what's
likely to happen.

Last March, for example, Sitaram Asur and Bernardo Huberman, two web scientists
from HP's Palo Alto lab, published a fascinating paper showing that the chatter on
Twitter can be used to forecast box-office revenue for movies. They analysed 2.89m
tweets by 1.2 million users referring to 24 different movies released over a period of
three months. They discovered that the rate at which movie tweets are generated can
be used to build a powerful model for predicting box-office revenue. They also found
that predictions derived in this way are "consistently better than those produced by an
information market such as the Hollywood Stock Exchange, the gold standard in the
industry".

To take just two examples, Transylmania, which opened on 4 December 2009, had the
lowest average tweet-rates (2.75 tweets per hour) of all the movies analysed and went
on to become the lowest-grossing opening for a movie playing at more than 1,000 sites,
making only $263,941 in its opening weekend. (It was subsequently pulled from cinemas
at the end of the second week.) At the other extreme, two movies that made big
splashes in their opening weekends – Twilight: New Moon (grossing $142m) and
Avatar ($77m) – had averages of 1,365.8 and 1,212.8 tweets per hour respectively.

guardian.co.uk/…/twitter-social-netwo… 1/3
23/10/2010 Social networks offer free access to th…
"Big deal," I hear you say. "Who – outside of Hollywood – cares about movie box-office
figures?" Well then, how about the stock market? Last week another interesting paper
appeared on arxiv.org, the stupendous online archive depository for research preprints
in physics, mathematics, computer science, quantitative biology, quantitative finance
and statistics. It's by Johan Bollen, Huina Mao and Xiao-Jun Zeng of the School of
Informatics and Computing at Indiana University at Bloomington, and it suggests that
data-mining on Twitter can significantly improve predictions of stock-market
movements.

Bollen and his colleagues set out to investigate whether public sentiment, as expressed in
large-scale collections of daily Twitter posts, can be used to predict the stock market.
They used a standard psychological tool to measure mood along six dimensions –
calmness, alertness, sureness, vitality, kindness and happiness – and then analysed
nearly 10m tweets posted by 2.7 million users between 28 February and 19 December
2008 to extract indicators of mood implicit in tweets. They then looked for evidence of
correlation between different kinds of mood and movements in the Dow Jones
industrial average.

They concluded that the predictive accuracy of standard stock-market prediction

models was significantly improved when certain mood dimensions were included (for
example "calm" and "happiness" as one measure of mood), but not others – "general
happiness" being one example.

Once upon a time, this kind of research would have attracted only the derisive snorting
of technophobes. Why should we take seriously the 140-character vapourings of geeks
and media types? What we really need to know is what "ordinary" or "normal" people
think. The problem with that kind of technophobic scepticism is that cyberspace is
increasingly coming to look like the "real" world. The demographic characteristics of
Twitter's 200 million users (or of Facebook's 500 million) are undoubtedly still a bit
skewed compared with those of the population as a whole, but over time that will
change: these social networks will increasingly be made up of folks like you and me. In
which case, the communications that go on between their members will become the best
guide we have to the zeitgeist.

If Ed Miliband wants to have a finger on the pulse of the nation, then the smartest thing
he could do would be to hire a few geeks and set them to work.

Ads by Google
Technical Analysis Alerts
Free Technical Analysis Newsletter Alerts On Penny Stocks About To Run
WhisperFromWallStreet.com/TA
Gold Prices Forecasts
Manage Risk & Exploit Opportunities in Commodities Markets, Get a Trial
BusinessMonitor.com/Gold
Oferta Tinta Original HP
Compre Tintas Originais da HP A partir de R$24,90. Aproveite!
www.HP.com/CartuchosOferta
Comments in chronological order (Total 0
comments)
There are no comments yet for this article.
In order to post a comment you need to be registered and signed in.
Register | Sign in
Post a comment

g u a r dia n . co. u k © Gu a r dia n New s a n d Media Lim it ed 2 0 1 0

guardian.co.uk/…/twitter-social-netwo… 2/3
23/10/2010 Social networks offer free access to th…

guardian.co.uk/…/twitter-social-netwo… 3/3