CITATION INDICES

Philip Purnell

March 2010
PART I
RESEARCH EVALUATION
 HOW DO WE EVALUATE RESEARCH?

• Research grants
– Number and value

• Prestigious awards
– Nobel Prizes

• Patents
– Demonstrating innovative research

• Faculty
– Number of post-graduate researchers

• Citation analysis
– Publication and citation counts
– Normalised by benchmarks
• Peer Evaluation
– Expensive, time consuming and subjective
DO WE NEED MORE METRICS?
A SIMPLE COUNT IS GOOD ENOUGH (1962)
 CHAMPIONS LEAGUE 2009
METRICS ARE HERE TO STAY !
Barcelona Team Statistics Manchester United
2 Goals 0
1 1st Half Goals 0
7 Shots on Target 2
4 Shots off Target 7
1 Blocked Shots 2
4 Corners 7
7 Fouls 10
1 Offsides 5
1 Yellow Cards 3
0 Red Cards 0
87 Passing Success 82.9
18 Tackles 21
72.2 Tackles Success 85.7
52.7 Possession 47.3
55.3 Territorial Advantage 44.7

5
 A BRIEF HISTORY OF THE CITATION INDEX
• Concept first developed by Dr Eugene Garfield
– Science, 1955
• The Science Citation Index (1963)
– SCI print (1960’s)
– On-line with SciSearch in the 1970’s
– CD-ROM in the 1980’s
– Web interface (1997) Web of Science
• Content enhanced:
– Social Sciences Citation Index (SSCI)
– Arts & Humanities Citation Index (AHCI)
• The Citation Index
– Primarily developed for purposes of information retrieval
– Development of electronic media and powerful searching tools have increased
its use and popularity for purposes of Research Evaluation
WEB OF SCIENCE
JOURNAL SELECTION POLICY

• Why do we select journals?

 THOMSON REUTERS
JOURNAL CITATION REPORTS
120

100 40% of the journals:

80 • 80% of the publications
% of database

60 • 92% of cited papers

40

20 4% of the journals:
0 • 30% of the publications
0 1000 2000 3000 4000 5000 6000
# of journals • 51% of cited papers
Articles Citations
COMPARE LIKE WITH LIKE
WHAT IS THE VALUE OF A CITATION?
• Why do people cite?
– Pay homage / give credit to pioneer
– Identifying a methodology
– Provide background reading
– Quotations
– Authenticating data, reproducing work etc
– Corrections
– Criticizing/Disclaiming someone's work/opinions

• Citations are an indicator of an article’s impact and usefulness to the

research community; they are the mode by which peers
acknowledge each other’s research.
• The value of a citation is only as important as its source.
– Clearly a citation from a prestigious peer review journal has more value than a
citation from non-scholarly material.
– How can you be sure that the citing source is reputable?

“When to Cite”, E. Garfield, Library Quarterly, v66, p449-458, 1996

WEB OF SCIENCE
JOURNAL SELECTION POLICY
• Approx. 2000 journals evaluated annually
– 10-12% accepted
• Thomson Reuters editors
– Information professionals
– Librarians
– Experts in the literature of their subject area

Journal ‘quality’
Web of Science

Journals under evaluation

THOMSON REUTERS
JOURNAL SELECTION POLICY
• Publishing Standards
– Peer review, Editorial conventions

• Editorial content
– Addition to knowledge in specific subject field

• Diversity
– International, regional influence of authors, editors, advisors

• Citation analysis
– Editors and authors’ prior work
Regional Journal Content Expansion 2007-2009
Contribution of Each Region to Subject Areas

Region A&H AB&ES CM EC&T LS PC&ES S&BS Total

AP 13 55 105 45 47 41 89 395
EU 150 88 161 81 50 116 200 846
LA 29 55 33 13 14 15 42 201
MA 13 23 32 10 15 7 31 131
NA 8 1 13 2 1 0 3 28
Total 213 222 344 151 127 179 365 1601

Regions: Subject Areas:

AP : Asia Pacific A&H : Arts &Humanities
EU : European Union AB&ES : AgBio & Environmental Sci
LA : Latin America CM : Clinical Medicine
MA : Middle East/Africa EC&T : Engineering Computing & Technology
NA : North America LS : Life Science
PC&ES : PhysChem & Earth Science
S&BS : Social & Behavioral Science
12
GROWTH IN COVERAGE OF
REGIONAL AND CZECH JOURNALS
Web of 2005 2010 Growth Variation
Science
Total 8,834 11,423 2,589 + 29%
Journals
Indexed

Regional 1,716 3,605 1,889 + 110%

Journals
Indexed

Czech 30 56 26 + 87%
Journals
Indexed

13
GROWTH IN COVERAGE OF CZECH
JOURNALS IN WEB OF SCIENCE
Web of 2005 2010 Growth Variation
Science
Science 23 36 13 + 57%
Citation
Index –
Expanded
Social 4 11* 7 + 175%
Sciences
Citation
Index

Arts & 3 12** 9 + 300%

Humanities
Citation
Index

*1 title is covered in both SCIE and SSCI 14

**2 titles are covered in both SSCI and A&HCI
GLOBAL RESEARCH REPRESENTATION
WEB OF SCIENCE COVERAGE

Region # Journals from Region in Web of Science

Europe 5,573 49%
North America 4,251 38%
Asia-Pacific 965 9%
Latin America 272 2%
Middle East/Africa 200 1%

Language # Journals in Web of Science

English 9114 81%
Other 2147 19%
GOVERNMENTS AND INSTITUTIONS
USING TR DATA FOR EVALUATION (INCL.)
• Czech Republic: Czech Academy of Sciences; Government
• France: Min. de la Recherche, OST - Paris, CNRS
• Germany: Max Planck Society, several gov’t labs, DKFZ, MDCUS:
National Institutes of Health
• United Kingdom: King’s College London; HEFCE
• European Union: EC’s DGXII(Research Directorate)
• US: NSF: biennial Science & Engineering Indicators report (since 1974)
• Canada: NSERC, FRSQ (Quebec), Alberta Research Council
• Australian Academy of Science, gov’t lab CSIRO
• Japan: Ministry of Education, Ministry of Economy, Trade & Industry
• People’s Republic of China: Chinese Academy of Science

16
EVALUATING COUNTRIES
CHANGING NATURE OF CZECH
RESEARCH
Czech researchers
averaged just over
4,000 papers a year in
the 70s – 90s.
Now over 10,000
papers per year

18
CHANGING NATURE OF CZECH RESEARCH
INTERNATIONAL COLLABORATION

Between 1970 and 1999, 14%

of Czech papers had at least
one international co-author

19
CHANGING NATURE OF CZECH RESEARCH
INTERNATIONAL COLLABORATION

In the last decade 2000 – 2010,

this figure has risen to 63%

20
SCIENTIFIC RESEARCH PRODUCTIVITY
IN CENTRAL EUROPE

21
Thomson Reuters InCites
SCIENTIFIC RESEARCH IMPACT
IN CENTRAL EUROPE

22
Thomson Reuters InCites
CZECH RESEARCH
BY OUTPUT AND IMPACT

23
Ranking by percentage share of Thomson Reuters (ISI) indexed papers 2004 - 2008
EVALUATING INSTITUTIONS (1)

EXTERNAL COMPARISONS
EVALUATING INSTITUTIONS

Number of citations
to North American
scientific papers

Source: Thomson Reuters

North America University Science Indicators
PUBLICATION AND CITATIONS
TOP CZECH ORGANISATIONS

26
COMPARE YOUR INSTITUTION’S
PERFORMANCE AGAINST GLOBAL PEERS

27
Thomson Reuters InCites
INSTITUTIONAL COMPARISONS
COMPARING IMPACT IN BIOLOGY?

Cambridge

Oxford

Stirling

29
Thomson Reuters InCites
INSTITUTIONAL COMPARISONS
IN SPECIFIC FIELDS

30
Thomson Reuters InCites
INSTITUTIONAL COMPARISONS
BIOLOGY PERFORMANCE

31
Thomson Reuters InCites
EVALUATING INSTITUTIONS (2)

INTERNAL ANALYSIS
WHAT IS OUR RESEARCH OUTPUT?

33
 BENCHMARKING YOUR PAPERS
AGAINST GLOBAL AVERAGES

This article is in the 12.92nd

Articles published percentile in its field by
in ‘Blood’ from Hematology articles from citations
2004 have been this year have been cited
cited 34.30 times 18.83 times

34
WHAT IS OUR RESEARCH OUTPUT?

35
 WHICH ARE OUR
CENTERS OF EXCELLENCE?

Physical Computer Science

Chemistry

41% below 220%

average above
average 36
Thomson Reuters InCites
WITH WHOM DOES OUR FACULTY
COLLABORATE?

37
WHICH OF THOSE COLLABORATIONS ARE
THE MOST VALUABLE?

38
INCITES
CITING ARTICLES LISTING

Thomson Reuters InCites

CITING PAPERS
IN THE FIELD OF PHYSICS

40
Thomson Reuters InCites
EVALUATING INDIVIDUALS
FRANTISEK VYSKOCIL
PROF. AND D.Sc CHARLES UNIV, PRAGUE
• Described: Potassium movement in the brain during migraine
• Member: Physiological Society of Cambridge and London
• Founding Member: Learned Society of the Czech Republic
• Awards:
– Prize of the Czech Academy of Sciences
– Lifelong Contribution Award of the Czech Academy of Sciences
– Purkynje Medal of the Czech Academy of Sciences

• First violinist: Quartet of the Herold Chamber Music Club

• Author: >60 popular science articles in "Vesmír“

42
 FRANTISEK VYSKOCIL
WEB OF SCIENCE PUBLICATIONS

247 Publications

43
FRANTISEK VYSKOCIL
GLOBAL INFLUENCE

Citations by country
Citations: 2.375
Cites per paper: 9,73
H-Index = 30
Citations by journal
WHO ARE OUR
MOST PRODUCTIVE AUTHORS?

45
Thomson Reuters InCites
WHO ARE OUR
MOST INFLUENTIAL RESEARCHERS?

46
Thomson Reuters InCites
WHICH AUTHORS HAVE
MOST IMPACT?

47
Thomson Reuters InCites
WHICH AUTHORS’ PAPERS HAVE
PERFORMED BEST IN THEIR FIELD?

48
Thomson Reuters InCites
HOW CAN WE COMPARE RESEARCHERS?

Author A: 60 papers Author B: 117 papers

49
Thomson Reuters InCites
EVALUATING JOURNALS
EFFICIENCY
JOURNAL IMPACT FACTOR

2005 2006 2007 2008 2009

All
Previous
Years

2008 Impact
Factor
Citations

Source paper – published in 2008

Cited reference – published in 2006 or 2007

CALCULATING 2008 IMPACT FACTOR
COLLECTION OF CZECHOSLOVAK CHEMICAL COMMUNICATIONS  

Citations in 2008
To items published in 2007 = 78
To items published in 2006 = 100
Sum = 178
178
= 0,784

Number of items 227

Published in 2007 = 118
Published in 2006 = 109
Sum = 227
JOURNAL IMPACT FACTOR
CZECH JOURNALS

53
CITATION BEHAVIOUR VARIES
BETWEEN SUBJECT CATEGORIES
JOURNAL 5-YEAR IMPACT FACTOR
CZECH JOURNALS

55
HOW DO CZECH INSTITUTIONS
EVALUATE JOURNALS FOR LIBRARIES?
• Faculty head request
• Publisher packages
• Budget constraints
• Library recommendation

56
IN WHICH JOURNALS DO OUR
BIOLOGISTS PUBLISH?

57
Thomson Reuters InCites
AND THE GLOBAL COMMUNITY
INFLUENCED BY YOUR RESEARCH?

58
Thomson Reuters InCites
SUMMARY:
THE DATA SOURCE
• Thomson Reuters solutions utilize the data from the Web of Science to
provide reliable and consistent evaluation tools
• It is essential to have a consistent, authoritative and clearly
defined body of data to create meaningful and dependable
statistics. This is why we use Web of Science:
– The gold standard citation resource, used by over 4,000 institutions in
more than 90 countries
– A comprehensive resource covering over 11,000 journals and thousands of
conference proceedings in the fields of Science, Social Science, Arts and
Humanities
– Unmatched retrospective depth of citation data (up to 110 years!)
– Selected content using unbiased, time tested and stringent journal
selection process
– Web of Science is also the source of the Impact Factor the most widely
accepted indicator of journal performance
PART II
THE DATA
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

61
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

62
CPCI – ON WEB OF SCIENCE

Search by
- Conference title
- Location
- Date
- Sponsor

63
WEB OF SCIENCE
CONFERENCE PROCEEDINGS CITATION INDEX
• Two Editions:
- Science & Technology
- Social Sciences & Humanities
• Approx 5.5 million records
• Updated Weekly
• Over 385,000 records per year
• More than 120,000 conferences covered in total
• Approximately 12,000 conferences covered annually
• Coverage from 1990, cited references from 1999
• Comprehensive, multidisciplinary and international coverage from
conferences proceedings published in books, journals, reports,
series, and preprints
• Details of each conference are indexed and searchable, data
includes: conference title, sponsor, location, description and date
 A combined set of results
from both Conference
Proceedings and Journal
literature.

Refine options are a useful tool to

narrow a search. For example narrow
by subject classifications or
document type to focus on the most
relevant materials
This Citation Report is based on the
entire publication portfolio of 2008
Nobel laureate Roger Y. Tsien.
Many records are from Conference
Proceedings, without this content the
portfolio is incomplete.
 Full Web of Science functionality
with Conference Proceedings
content

Details of both
conference and
source
publication

Classified by
Document Type

Source item is assigned to

standard subject
classifications
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

69
 WEB OF SCIENCE
DEPTH OF CITATION INDICES

1900 Century of Science 1945 Science Citation Index

1900 Century of Social Sciences 1956 SSCI
1975 A&HCI
56 years of added coverage 1990 CPCI
in the Social Sciences

>300 journals added that

focus on the first half of
last century
CENTURY OF SOCIAL SCIENCES
BREAKDOWN BY FIELD
CoSS distribution by field Title Count
ANTHROPOLOGY 24
COMMUNICATION 18
ECONOMICS & MANAGEMENT 47
EDUCATION 31
GEOGRAPHY 12
HISTORY & PHILOSOPHY OF THE SOCIAL SCIENCES 1
LAW 30
POLITICAL SCIENCE 10
PSYCHIATRY 21
PSYCHOLOGY 65
PUBLIC HEALTH 20
SOCIAL ISSUES & SOCIOLOGY 29
Total 308
COVERAGE IN THE FIELD OF GEOGRAPHY
• There are more than 40,000 records in the field of Geography
between the years of 1900 and 1955
• The key journals from this period have been evaluated, sourced and
indexed to provide a unique resource:
– Geographical Journal – Petermanns Geographische Mitteilungen
– Scottish Geographical Magazine – Economic Geography
– Geography – Annals of the Association of
– Geographical Review American Geographers
– Annales de Geographie

• The works of many ground breaking researchers from around the

world have been included, such as:
– C. Warren Thornthwaite
– Harlan H. Barrows
– John R. Borchert
CASE STUDY:
Charles Warren Thornthwaite
• C. Warren Thornthwaite was a Geographer
and Climatologist best known for devising
a Climate Classification System proposed
in his landmark paper in 1948 which also discusses the
importance of evapotranspiration on climate.
• Thornthwaite published 28 papers between 1931 and 1965,
mostly in the fields of Geography and Geochemistry &
Geophysics.
• His works have gone on to be highly influential to researchers
in many fields and in recent years have been particularly
utilized by researchers involved in climate change and water
management
The works of C. Warren Thornthwaite

Thornthwaite
produced 28 highly
influential works in the
Social Sciences which
have been
consistently cited
throughout their
history
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

76
ISI WEB OF KNOWLEDGE 5
webofknowledge.com

77
ISI WEB OF KNOWLEDGE 5
CITATION UNIVERSE

78
ELIMINATION OF STOPWORDS

79
LEMMATIZATION

run, runs, ran and

running are forms of
the same lexeme, with
run as the lemma.

80
FULL RESULT COUNTS
NO MAXIMUM ON ‘ANALYZE RESULTS’

81
LEFT-HAND TRUNCATION

All users are

accustomed to using
right-hand truncation in
their searches

82
LEFT-HAND TRUNCATION

Now with left-hand truncation,

searches can be even more
accurately defined

83
ABSTRACT PREVIEW

84
ABSTRACT PREVIEW

85
CITATION COUNTS USING
WEB OF KNOWLEDGE VALUES

86
WEB OF KNOWLEDGE
CITATION BREAKDOWN TABLE

87
MARKED LIST ENHANCEMENTS
- ANALYZE TOOL & CITATION REPORTS
Analyze tool and citation
report avaiable for specific
products. Future release will
include these features for the
All Databases results

88
CITED REFERENCE SEARCHING
NOW FOR ALL CITATION INDICES

New cited reference search

parameters facilitate a more
accurate search

89
CITATION COUNTS IN FULL

90
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

91
BIOSIS CITATION INDEX
BIOSIS CONTENT WITH CITATION DATA
• Biosis Previews
– Widest collection of life sciences fields in the world
– agriculture, biodiversity, biotechnology, botany, drug
discovery, gene therapy, marine biology, wildlife
conservation, zoology
• Biosis Citation Index
– Provides cited references for BIOSIS Previews
• Cited references for all Previews content (beginning 2006)
• Cited references for 60% of high impact Previews journals
(1926 - 2005)

• Exclusive to ISI Web of Knowledge platform

BIOSIS CITATION INDEX
ON ISI WEB OF KNOWLEDGE 5

93
BIOSIS CITATION INDEX
SEARCHING

Set your search

parameter defaults
Select years, use
lemmatization and sort
by publication date

94
BIOSIS CITATION INDEX
REFINE RESULTS
Refine results or exclude
Identify the most abundant results from your dataset
sources of relevant results via a series of parameters
or concentrate on harder-
to-find document types

95
 BIOSIS CITATION INDEX
SORTING AND EXPORTING RESULTS
Sort your results to find
breaking new research papers
or to identify the most
influential or relevant papers
Select the records of your in your field
choice and print, email or
export them via your favourite
bibliographic management
system

96
BIOSIS CITATION INDEX
MARKED LIST

97
BIOSIS CITATION INDEX
CITATION BREAKDOWN TABLE

98
BIOSIS CITATION INDEX
CITATION REPORT

99
BIOSIS CITATION INDEX
ANALYZE RESULTS

Analyze by Super Taxa

100
BIOSIS CITATION INDEX
CITATION MAP
Citation maps can find
relevant papers not returned
in your search results

2nd generation
citations reveal
citation pathways

101
BIOSIS CITATION INDEX
CITED REFERENCE SEARCHING

Cited reference
searching helps users
to find ‘missing
citations’
102
 BIOSIS CITATION INDEX
SPECIALIZED INDEXING

Searching Biosis Citation Index for

‘Fossil* and Africa and humans’
returns this paper

The same search in ‘All

Databases’ reveals that this record
is also present in Web of Science

But the same search in Web of

Science does not find the article

It is the Biosis specialized indexing

fields that has found these terms
that are not indexed in Web of
Science

103
BIOSIS CITATION INDEX
SPECIALIZED INDEXING
• All BIOSIS records are indexed by biology specialists using
the following unique fields

Major Concepts Concept Code(s)

Taxonomic Data Disease Data
Chemical Data Gene Name Data
Sequence Data Geographic Data
Geologic Time Data Methods and Equipment Data
Parts & Structures Data Miscellaneous Descriptors

104
 BIOSIS CITATION INDEX
SPECIALIZED INDEXING
All the specialized indexing
fields are searchable

The concept code 00506

codes for Explorations and
Expeditions

This paper is returned

because the concept code is
found on the BIOSIS record

105
BIOSIS CITATION INDEX
SUMMARY
• Combines
– World-leading Biosis life sciences content
– with valuable cited references back to 1926

• Enhances
– Web of Science searches via the All Database Search
using common indexing backbone

• Employs
– ISI Web of Knowledge cited reference searching

• Exclusive
– ISI Web of Knowledge

106
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

107
EMERGENCE OF CHINESE RESEARCH

108
CHINESE SCIENCE CITATION DATABASE
AVAILABLE NOW!
• Over 2 million records
• Over 13 million cited references
• Subject Areas
– Mathematics, Physics, Chemistry, Geosciences, Biology,
Agricultural Sciences, Engineering (Computer Science,
Mechanics, etc.), Medicine, and Management

• Dual Language
– Since 2002, 80% of records have full English bibliographic
information and most have English abstracts

• Content is searchable in both English and Chinese

110
CSCD
CITATION BREAKDOWN TABLE

Bibliographic
information in
English and
Chinese
111
THE DATA

• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences

• ISI Web of Knowledge - The Citation Universe

– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database

• Research Analytics

112
AUTHOR NAMES

113
AUTHOR NAMES

114
RESEARCHERID
SCHOLARLY RESEARCH COMMUNITY

• Accurate Identification
• Organize and Manage
• Increase Visibility and Recognition
• Measure Performance
• Collaboration
• Security

Science, March 2009

 INDIVIDUAL LEVEL
RESEARCH EVALUATION

See personalized
metrics using Web
of Science citation
data

View accurate publication

list due to unique author
identification
RESEARCHER ID
ANALYZE COLLABORATION NETWORK

Seek global collaboration

opportunities by author,
field, institution or country
RESEARCHER ID
VISUALIZE CITING ARTICLES NETWORK
RESEARCHER ID
ELECTRONIC CV RESOURCE

ResearcherID Profile:
http://www.researcherid.com/rid/A-
9180-2008
RESEARCHER ID
GLOBAL PARTICIPATION
Top Ten Institutions
TopCollege
University Five Institutions
London Top Five Countries
The University of Queensland
Monash University
Harvard University
University of Michigan
University of Pennsylvania
ETH Zurich
University of Cambridge
Stanford University
McGill University
RESEARCHERID
UPLOAD SERVICE

• Institutions can upload content on behalf of their

researchers
– Upload researcher names  Obtain a ResearcherID
account
– Upload individual publication portfolios  Articles are
matched to Web of Science records and ResearcherID
portfolios are generated
RESEARCHERID
DOWNLOAD SERVICE

• Download data about the individuals at your

institution
– Names and name variants, current and past affiliations

• Download ResearcherID publication portfolios

– Bibliographic details of each item in the portfolio
– For articles that were successfully matched to Web of
Science records there will be a Times Cited count and UT
tag.

• Can be used by the customers for their own

internal systems