Benchmarking science:

a critique of the ISI (Thomson-Reuters) index

C. K. Raju
School of Mathematical Sciences
Universiti Sains Malaysia
1170 Penang, Malaysia
1. Introduction
he !rgani"ation of #slamic Conference $eci$e$
1
in %ec &00' to urge Mem(er States to strive for
)*uality e$ucation that +romotes creativity, innovation an$ research an$ $evelo+ment., he -ision
1..1 on Science an$ echnology a$o+te$ (y the 10th session of the #slamic summit in Putrajaya,
&

Malaysia, in &00/, ha$ alrea$y $eclare$ that )!#C mem(er states are committe$ to (ecome a
community that values 0no1le$ge an$ is com+etent in utilising an$ a$vancing S2 to enhance...socio3
economic 1ell3(eing,. his is in$ee$ a lau$a(le o(jective.
4o1ever, noting the lac0 of o(jective metho$ologies, to evaluate +erformance of universities, the !#C
$eci$e$ to evolve one. 5 meeting of technical e6+erts in ehran in &007 $rafte$ a $ocument
containing the +ro+ose$ criteria for ran0ing of universities in the !#C region. his $eci$e$ to a$o+t a
)0ey +erformance in$icator, 1hich gives '07 1eightage to research *uality an$ out+ut. he em+hasis
on research *uality is again a 1orth1hile an$ lau$a(le move.
/
he $ifficulty (egins 1ith the criteria use$ to measure research *uality. he echnical %ocument
asserts 8+. 119 that )it is internationally a$mitte$ that the *uality of a given +u(lishe$ article can (e
measure$ (y the num(er of citations it receives,. #t also states 8+. 109 that )accor$ing to the
international stan$ar$s, only the journals classifie$ (y the #nstitute of Scientific #nformation
8#S#9...shoul$ (e use$ (oth for counting +u(lications an$ citations,.
hus the 1ay to ju$ge research out+ut recommen$e$ (y !#C is 8a9 +u(lication 8in #S# journals, others
not counte$9, 8(9 citation in$e6, 8c9 im+act +arameter 8e6clusiveness of journal9. his has no1 (een
1i$ely im+lemente$ in various universities across the #slamic 1orl$. Research fun$ing an$ +romotions
in universities are $eci$e$ on the (asis of this assessment.
1 !#C Summit, en :ear Strategic Plan, +u(lic statement, Third Extraordinary Session of the Islamic Summit Conference,
Makkah al Mukarramah, Kingdom of Saudi Arabia, 5! dhul "a#dah $%&! h '( )ecember &**5, sec -.1.+
& !rganisation of #slamic Conference, Conference on Science an$ echnology, -ision 1..1; Kuala <um+ur %eclaration
on Science an$ echnology for the Socio3=conomic >ell3?eing of the Ummah, 7310 !cto(er &00/.
/ !#C Re+ort. 8&0079. Re+ort on a$o+te$ criteria, +roce$ures an$ mechanisms for ran0ing of universities. ehran, 5+ril
&007. htt+;@@111.oic3oci.org@oicne1@+oa@ran0ing.+$f.
2. Objectives vs methodology
4o1ever, as argue$ in this +a+er, the +ro+ose$ metho$ology conflicts 1ith the state$ o(jectives. #n this
situation, it is the metho$ology 1hich 1ill (e follo1e$ (y +eo+le at the groun$ level, not the state$
o(jectives. hus, the criteria of measurement, not the state$ o(jectives, 1ill $etermine the actual
outcome, 1hich 1ill (e contrary to the state$ o(jectives.
5ccor$ingly, the criteria for measurement of research *uality an$ its e6act relation to the state$
o(jectives nee$s to (e carefully re3e6amine$ to ensure that the state$ o(jectives are met.
5ccor$ing to the state$ o(jectives, research is (eing encourage$ to su++ort socio3economic 1ell (eing
of the community. his means that research *uality must (e measure$ (y the socio3economic 1ell
(eing it (rings a(out.
Does citation index correlate with socio-economic well-eing!
?ut no(o$y ever $emonstrate$ that the citation in$e6 8counte$ using #S# journals9 correlates 1ith socio3
economic 1ell (eing, es+ecially in non3>estern societies. here is not a single stu$y or +u(lication on
this. So, this cannot (e acce+te$, even if it is )internationally a$mitte$,. Counter e6am+les are easy to
fin$. Aor e6am+le, there are thousan$s of +a+ers +u(lishe$ on string theory. Many among these are also
highly cite$. 4o1ever, there is no im+act of this research on the socio3economic 1ell (eing of anyone
8other than the researchers themselvesB9 for the +ast thirty years.
Is citation index a measure of truth!
here is no scientific evi$ence that the citation in$e6 is even a relia(le measure of truth.
hus, thousan$s of 1ell3cite$ +a+ers have (een +u(lishe$ on $ar0 matter in the gala6y an$ its
com+osition, though it no1 seems li0ely that this $ar0 matter is an artificial hy+othesis, an$ it is
unli0ely to e6ist in the gala6y 8in significant amounts9. here are many other cases 1here highly3cite$
)fashiona(le, 1or0s sim+ly 1ent out of fashion.
Science is full of sur+rises; for eighty years scientists uniformly (elieve$ the e6+ansion of the cosmos
1as $ecelerating, an$ routinely s+o0e of the $eceleration +arameter, (ut they no1 (elieve that its
e6+ansion is accelerating $ue to something un0no1n calle$ )$ar0 energy,. his is +art of the +resent3
$ay )stan$ar$ mo$el, of cosmology. hey may again (e 1rong. hus, the citation in$e6 only measures
+o+ular o+inion among certain scientists, 1hich +o+ular o+inion can (e fic0le, 1hereas socio3economic
1ell3(eing is something more soli$ an$ not so easily u+set. So, it is unli0ely that a firm relation 1ill
e,er (e esta(lishe$ (et1een citation in$e6 an$ socio3economic 1ell (eing.
"itation index correlates negatively with well-eing in the non-#est
=ven if it is grante$, for the sa0e of argument, that there may (e some sort of general relation (et1een
citation in$e6 an$ the socio3economic 1ell3(eing of >estern societies, the same is not necessarily true
for non3>estern societies. !n the contrary, it is easy to see many situations 1here the citation in$e6
correlates negati,ely 1ith the socio3economic 1ell3(eing of non3>estern societies. hus, at a training3
seminar in Malaysia on ho1 to ma6imi"e citations, the trainer re+orte$ly +ointe$ out that +eo+le shoul$
not 1or0 on $iseases of local significance to Malaysia, (ut shoul$ 1or0 on $iseases of international
significance. he same com+laint has (een hear$ in #n$ia, that researchers $o not 1or0 on $iseases li0e
malaria, an$ tu(erculosis, relevant to #n$ians, (ut instea$ research issues li0e heart3$isease, 1hich are
of greater interest in the >est.
5 s+ecific e6am+le is that of )#n$ian chil$hoo$ cirrhosis,. >hole families use$ to suffer from the issue
of juvenile cirrhosis of the liver, lea$ing to numerous early $eaths. ?ut no(o$y (othere$ to stu$y it,
(ecause this 1as a s+ecifically #n$ian +ro(lem. he issue 1as clarifie$ 1hen steel +rices 1ent $o1n,
an$ +eo+le s1itche$ to stainless steel vessels for coo0ing. he +henomenon $isa++eare$, so the reason
(ecame clear. he earlier technology 1as that of using a metal calle$ )+ital, 8(rass9. hough there 1as
a s+ecific tra$itional techni*ue of tinning the surface 8calle$ )0alai,9, carelessness in this le$ to those
$eaths. 4o1ever, no researcher (othere$ to +oint this out, just (ecause this 1as not of )international
significance,.
$ther deficiencies of the citation index and im%act %arameter
Many other arguments can (e (rought to (ear against the citation in$e6. #t 1as the +ro$uct of a one3
man start3u+ com+any 8#nstitute of Scientific #nformation9, 1hich +ro$uct 1as successfully mar0ete$ to
large sections of the scientific community. #t has no1 (een ta0en u+ (y one of the largest +u(lishing
conglomerates; homson3Reuters, (ecause this criterion clearly hel+s to increase the sales of scientific
journals, in general, an$ their science journals in +articular.
Caturally, this for3+rofit com+any is +rimarily concerne$ a(out its +rofits, not scientific truth. #f there is
a conflict of interests it is not $ifficult to imagine 1hich 1ay the com+any 1ill choose. o entrust
scientific truth in the han$s of a single for3+rofit com+any is $angerous. o allo1 it to ta0e $ecisions
secretively is 1orse; the $ata on 1hich the im+act +arameter is (ase$ is not +u(lic, an$ not share$ (y
homson Reuters 1ith researchers.
.
%ou(ts have (een e6+resse$ that the com+any may mani+ulate
matters for +ur+oses of +ro+agan$a or on +ayment.
'

5+art from ethical issues, there are (asic statistical issues. hus, one journal, in an e$itorial, cite$ all
the +a+ers +u(lishe$ in that journal. his cause$ its im+act +arameter to jum+. he $istri(ution is
s0e1e$, an$ may 1ell (e long3taile$, so it is not clear 1hat is the *uality of the estimators use$.
he 0ey3+erformance in$e6, +ro+ose$ (y the !#C has unfortunately neglecte$ all this criticism, just (y
saying that it is )internationally a$mitte$, that citation is the measure of *uality.
4o1ever, a much more serious matter, 1hich ma0es the citation in$e6 +roce$urally (a$, an$ damaging
to the interests of non3>estern countries, is the follo1ing.
. M. Rossner, 4. -an =++s, an$ =. 4ill, )Sho1 me the $ata,. -+ Cell .iology 179 8D9, ++. 10E13E&.
' he P<oS Me$icine =$itors. &00D. he im+act factor game. #t is time to fin$ a (etter 1ay to assess the scientific
literature. P<oS Me$. /;e&E1 $oi;10.1/71@journal.+me$.00/0&E1
3. The criterion of reputability
<ong ago Po++er, in his /ogic of Scientific )isco,ery +ro+ose$ the criterion of falsifia(ility, 1hich he
later
D
rename$ to the refuta(ility, saying he $i$ not 0no1 =nglish 1ell enough 1hen he gave the first
name in 1E/'. his criterion of refuta(ility $oes have certain merits, for it $eman$s that scientific
0no1le$ge must (e continuously an$ rigorously +ut to test.
#n contrast, +arameters such as the citation in$e6 an$ im+act +arameter only measure re0utabilityF this
$oes not even guarantee goo$ science 1hich must (e refuta(le, not re+uta(le. Aor e6am+le, +eo+le have
com+laine$ that many results of string theory are re+uta(le, (ut not refuta(le. hat is they are )not even
1rong,,
7
so one can learn nothing from the rejection of these theories. he case of the stea$y state
theory is similar.
Re%utailit& as a means of social control: %reser'es an& elief
!n the other han$, it is 1ell 0no1n that for centuries the church use$ this criterion of re+uta(ility as a
means of social control. #t sim+ly $eclare$ all its critics as $isre+uta(le )heretics, etc., to +revent its
(eliefs from (eing challenge$. hus, 1e have a long (aseline of em+irical evi$ence to sho1 that this
criterion of re+uta(ility is (a$ for the truth, for it ena(le$ the church to +reserve the most a(sur$
conclusions.
5 1ell30no1n e6am+le is the laugha(le $ate of creation of the 1orl$ 8&/ !cto(er .00. ?.C., at nine
oGcloc0 in the morning9 as $etermine$ 1ith e6*uisite +recision (y that most re+uta(le of 1orthies the
then -ice Chancellor of the University of Cam(ri$ge, Sir Hohn <ightfoot,
I
1ho +u(lishe$ it in 1D..,
(efore ?isho+ Ussher. his 1as (roa$ly (elieve$ in the >est for some 1' centuries, an$ >estern
scholars in the 1E
th
c. calle$ #n$ians )su+erstitious,, hence $isre+uta(le, for (elieving the cosmos 1as
much ol$er. hus, the criterion of re+uta(ility, measure$ (y the citation in$e6, encourages social
control, not truth or goo$ science. #t can lea$ to an$ +reserve the most a(sur$ conclusion. his much
has (een em+irically $emonstrate$ over a long +erio$ of time.
his +henomenon is not restricte$ to theology. =instein 1as the most re+ute$ scientist in the last
century, an$ his false claim to the theory of relativity hel$ u+ the $evelo+ment of science, an$ the use
of functional $ifferential e*uations.
E
Scientists are afrai$ of 1riting anything against scientific
authority; just as the church name$ $issenters heretics, to arouse +reju$ice against them, $issenting
scientists get la(elle$ )cran0s,. Hust as the church never $e(ate$ 1ith those it $eclare$ heretics,
scientists $o not $e(ate 1ith )cran0s,. Aor e6am+le, the 1ell 0no1n re+ository 5rJiv has a secretive
system of such classification.
D Karl Po++er, 1ealism and the Aim of Science, PostScri+t to <Sc%, vol. 1, 4utchinson, <on$on, 1EI&.
7 Peter >oit, 2ot E,en 3rong4 The failure of string theory and the search for unity in 0hysical la5, ?asic ?oo0s, &00D.
I 5. %. >hite, The 3ar of Science and 1eligion, 5++leton an$ Co., 1IE7, +. E..
E his is too com+licate$ an issue to go into here. Aor the (asics of functional $ifferential e*uations, see C. K. Raju, Time4
To5ards a Consistent Theory, Klu1er 5ca$emic, %o$recht, 1EE.. Aor an account for a lay+erson, see C. K. Raju, The
Ele,en 6ictures of Time, Sage, &00/. Aor a solution of functional $ifferential e*uations in a significant +hysical conte6t,
see, C. K. Raju, )he electro$ynamic &3(o$y +ro(lem an$ the origin of *uantum mechanics,.7ound+ 6hys. 34, &00., ++.
E/7KD&. Aor more on =insteinGs mista0e, see my acce+tance s+eech for the L5 a1ar$, htt+;@@c0raju.net@Ce1s@c0r3L53
acce+tance3s+eech.+$f. .
Re%utailit&: in whose hands is the social control!
he imme$iate +oint, ho1ever, is a $ifferent one. Since the criterion of re+uta(ility serves as a means
of social control, the *uestion is this; in 1hose han$s is that social controlM
=ven a cursory e6amination of the homson3Reuters 8formerly #S#9 $ata(ase sho1s that the list of
journals in it is +re$ominantly >estern 8i.e., (ase$ in US, Cana$a, UK, =uro+e an$ 5ustralia, Ce1
Nealan$9. =*ually, the e$itorial (oar$s of these journals are +re$ominantly >estern. Aor e6am+le, there
are 1'E journals relate$ to e$ucation, all (ase$ in the >est. !f the thousan$s of e$itorial a$visors in
these journals only & 80.&79 are #n$ian, an$ this ethnicity counts for nothing since these & are (ase$ in
the >estB 4ence, even if 1e grant that +u(lication in #S# journals is someho1 suite$ to assess research
*uality for >estern researchers, this is not the case for non3>estern researchers, for >esterner +lay the
role of su+eriors, not +eers.
hus, any researcher 1ho 1ants to change the e$ucation system in #n$ia must first gather )cre$i(ility,
or re+uta(ility (y going 1ith a (egging (o1l to these >esterners 8very conscious of their +olitical an$
socio3economic $ominance9 to +u(lish his or her i$eas in those journals. his ensures status *uo, for
nothing can (e change$ unless it is re+uta(le, an$ >esterners have (een given the authority of censors
to $eci$e 1hat is re+uta(le, just as the church $eci$e$ it in the >est not long ago. hus, the use of the
citation in$e6 +rovi$es a stic0 an$ means of social control to the >est over the non3>est.
hat is, the measurement of research *uality through +u(lications in these journals serves to +ut non3
>estern scientists un$er the control of >estern scientists. %oes that im+rove their socio3economic
1ell3(eingM
8#t 1as in this conte6t that the USM recently organi"e$ a conference on )%ecolonising our
Universities, 1hich sought to change the curriculum in our universities, across the (oar$, inclu$ing in
mathematics an$ science. his ha$ a significant im+act on +eo+le, an$ may result in a variety of +olicy
changes, 1ith a much greater long3term im+act. ?ut this 1oul$ count for nothing on the KP#, (ecause it
is $one outsi$e the control of the >est.9
"reating a class de%endent on the #est
4o1ever, there is one sort of socio3economic 1ell3(eing 1ith 1hich the $eman$ for #S# +u(lications
does correlate; in fact there is an insi$ious causal relationshi+. 5s a measure of >estern re+uta(ility
this is an even more $angerous conse*uence. #t creates a class of intellectuals an$ aca$emics 1ho are
allie$ to the >est, an$ $e+en$ent u+on its goo$1ill for their material +ros+erity. 5 han$ful of ?ritish
1ere a(le to rule #n$ia +recisely (y creating such a class 1ith veste$ interests, (y means of the
e$ucational OreformsO of Macaulay.
10
he long3term conse*uence has (een that #n$ia, an e6+orter of
0no1le$ge an$ technology to ?ritain until the 1Ith c., turne$ into a chronic im+orter of 0no1le$ge an$
technology for the ne6t t1o centuries, an$ remains so to this $ay, in a state of $e+en$ence. hus, use of
the citation in$e6 hel+s to create a class of $e+en$ent intellectuals 1ho hel+ the socio3economic 1ell3
(eing of the >est at the e6+ense of the non3>est.
#n$ee$, since these )>estern3re+uta(le, +ersons are invaria(ly selecte$ as )e6+erts, for various
aca$emic tas0s such as $eci$ing the curriculum, this means that various elements of in$octrination,
10 C. K. Raju, Ending Academic Im0erialism4 a .eginning, Multiversity an$ Citi"ens #nternational, Penang, &011.,
such as a false >estern history of science can (e easily smuggle$ into the sylla(us, as has (een $one in
#n$ian school te6ts.
11
his also means that 1hole generations gro1 u+ in$octrinate$ into a1e of the
>est 1ith all sort of falsehoo$s. =ven more $eviously, these >estern3aligne$ e6+erts have +rescri(e$
the teaching of formal mathematics 8as o++ose$ to scientific mathematics9 1hich involves
in$octrination into a (iase$ form of meta+hysics 1hich is o++ose$ to Muslim, ?u$$hist, an$, in$ee$,
all non3>estern +hiloso+hy.
1&
5ll this is a 0ey source of soft +o1er to ensure >estern $ominance.
#t seems the !#C countries have incorrectly acce+te$ a metho$ology 1hich ensures >estern
$ominance, e6actly as #n$ia 1as conne$ into acce+ting the >estern mo$el of e$ucation through a false
history of science. 8he carrot then 1as the same; a$vancement in science an$ technology.9 #f the
current )0ey +erformance in$icators, are not imme$iately revie1e$, they are li0ely to usher in a long
an$ $isastrous +erio$ of $e+en$ence on an$ $omination (y the >est.
3. Why is journal publication important?
=ven if all these consi$erations are set asi$e, it is astonishing that a large grou+ of nations has $eci$e$
to follo1 the claims of a small for3+rofit com+any 8#S#9 no1 sol$ to a large for3+rofit com+any
8homson3Reuters9, 1ithout $ue $iligence. >hile there is no correlation of the citation in$e6 1ith the
socio3economic 1ell3(eing of the 0eo0le in the !#C countries, there is a clear correlation of the citation
in$e6 an$ im+act +arameter 1ith the socio3economic 1ell3(eing of these com+aniesB #n$ee$, the entire
(usiness of #S# 1as (uilt aroun$ this +ro$uct. #t is clear 1hy the largest +u(lishing com+any 8homson9
1oul$ 1ant to encourage the i$ea that the *uality of scientific research can (e measure$ only (y
0ublications, that too in a list of journals, the majority of 1hich are (rought out (y large +u(lishing
houses. hese +u(lishing houses ma0e a huge +rofit (y means of a various government su(si$ies.
Airst, the costs of research are (orne (y the state, (ut the o1nershi+ of the outcome 8+u(lications9 vests
in these com+anies to 1hom co+yright of the +u(lications is transferre$. he refereeing etc. is also
$one for free (y the aca$emic community, an$ these $ays even the ty+esetting an$ +roof3correction is
often $one (y the author himself. 4o1ever, the journals ma0e a huge +rofit (y charging for the journal,
an$ for re+rints etc. hus, governments +ay t1ice; first to generate the 0no1le$ge, an$ then to (uy it
(ac0 from the +u(lishers. he +u(lishers invest nothing to ma0e a +rofit.
#t is not clear 1hat e6actly is the service these +u(lishing houses +erform, in return for 1hich they get a
+rofit, since to$ay $istri(ution of information is mostly $one electronically, an$ fe1 +eo+le actually
rea$ +rinte$ journals. #t is one thing that the results of scientific research are (etter off +u(licly
$isseminate$. ?ut this coul$ (e $one in a variety of 1ays; (y sim+ly tal0ing to others, as in a
conference, or (y 1riting u+ the results an$ +osting them on a +re+rint server, etc. #n$ee$, a Aiel$s
me$al 1as recently a1ar$e$ for a +iece of research 1hich 1as never +u(lishe$. !nce u+on a time the
journal 1as im+ortant (ecause it too0 oneGs results to a 1i$er community. hat is no longer the case,
an$ journals no longer +erform that role. !n the contrary, +u(lication in a journal restricts the
rea$ershi+ to those 1ho can affor$ to +urchase the journal.
he journals 1ill maintain that they +rovi$e the service of *uality control. hat is not really true. >hat
the journals $o is to sen$ a su(mitte$ +a+er to referees. he tas0 of *uality control is +erforme$ (y the
referees. Co1 this is a secretive +rocess. 5s such, there is no o(jective 1ay to ju$ge the *uality. >e
11 C. K. Raju, )eaching racist history,, Indian -ournal of Secularism 118.9 8&00I9 &'3&I.
1& C. K. Raju, )eaching mathematics 1ith a $ifferent +hiloso+hy. 1; Aormal mathematics as (iase$ meta+hysics, 8to
a++ear9 Science and Culture, 77 873I9, &011, ++. &7'3I0. 8#nci$entally, this is an #S# journalB9
sim+ly have to (elieve in the e$itor. Aor all one 0no1s, the e$itor might intro$uce a variety of (iases,
an$ some e$itors may (e $o1nright $ishonest. his 1as ho1 the system of secretive censorshi+
functione$ 1ith the church. !ur (elief in the *uality control of +rovi$e$ (y the journal or censorshi+
thus rests on mere faith in the e$itor.
. !o "hat is the alternative?
Clearly, in to$ayGs electronic age, journals have (ecome re$un$ant for +ur+oses of $issemination of
scientific 0no1le$ge, 1hich can (e *uic0ly an$ efficiently +erforme$ electronically. Secretive
+rocesses cannot ensure *uality in science.
5ccor$ingly, a sim+le solution to the matter is to have 0ost0ublication 0ublic debate. #nstea$ of
+u(lishing a +a+er, an author sim+ly +osts in on a +u(lic +re+rint server. he e$itor, electe$ (y the
community 8not +u(lishing house9 invites referees to revie1 the article +u(licly. he author gets a
chance to res+on$ +u(licly. Peo+le can rea$ (oth sets of comments an$ ma0e u+ their o1n min$. he
referees 1ho no1 1or0 for free for +u(lication houses, 1ill (e $oing the same 1or0 no1 for the 1hole
community. his 1oul$ ma0e the results of scientific research availa(le to all, not merely those 1ho
can +ay for journals. 5s suggeste$ in my (oo0 on Ending Academic Im0erialism, one nee$s to start
such a Aree Science Aorum, some1hat li0e the Aree Soft1are Aoun$ation. 4o1ever, since those
running the forum $onGt aim to ma0e a livelihoo$ out of it, it 1oul$ re*uire very little investment, an$
the !#C countries 8an$ in$ivi$ual universities9 can easily set u+ +re+rint3servers to ena(le this. his
1oul$ (e a soli$ means to com(at the +ernicious influence of reliance on (oth the >estern criterion of
re+uta(ility an$ large +u(lishing houses.