On Writing ICTD Research Papers

Kentaro Toyama (Microsoft Research India; kentoy@microsoft.com)

Co-Founder, IEEE/ACM International Conference on Information and Communication Technologies and Development
Last updated: September 3, 2008

Below are some high-level comments on writing ICTD research papers, which form the core content of the
IEEE/ACM International Conference on Information and Communication Technologies and Development.

As with any field of study, what work constitutes the field, and how quality is judged, remain open
questions. The field, of course, is still evolving and its standards will be determined by the community as a
whole. What makes ICTD research research, and not just reports about technology projects in rural
villages? What does it mean to have economics and anthropology research side by side with computer
science and electronic engineering? How can we preserve the integrity of methodologies developed by
individual disciplines, while allowing for cross-fertilization among fields and a healthy respect for other
disciplinary perspectives?

Having served the conference in various capacities, I’ve asked these questions from a variety of
perspectives, some of it in heated discussion with colleagues. Below is a sketch of what I arrived at for
authors, particularly in terms of what constitutes good ICTD research and what goes in a good ICTD paper.
It undoubtedly reflects my personal biases. Nevertheless, my hope is that the ideas below provide some
guidance for those new to ICTD research or to authors submitting papers to the ICTD conference.

Research in General
Some characteristics of good research transcend disciplines, and I’ll start with what is required for good
research in most fields. I believe these apply to anthropology, sociology, economics, electrical engineering,
design, etc.

Novelty: Some aspect of the research has not been done before. It might be a new thesis, fresh data,
innovative methodology, a novel invention, a first-time evaluation, a first good evaluation, and so forth. Of
course, it’s not necessary to be new in every way, but something in the work should extend the state of
knowledge in the world. Lack of novelty, at least relative to existing literature, almost immediately
disqualifies a body of work from acceptance to research publications, however well executed.1 So, in a
paper, it helps to make the novel contributions crystal clear – summarized in the abstract, explained in the
introduction, and recapitulated in the conclusion.

Applicability: The research solves or answers an existing or anticipated problem or question.2 Just for
example, “interventions” are applicable by definition, but they should still demonstrate their capacity to
solve at least some of the challenges attempted. Observational work should support decision making about
development. And, so on. In ICTD papers, how a particular research project is applicable to development is
worth mentioning explicitly, probably in the introduction, and then in some detail in a later analysis or
evaluation section.

Generalizability: The research contributes to a body of knowledge that applies to a set of circumstances
beyond the specific instance of the actual work. 3 Technologies are generalizable if they work under a

1
Novelty is relative, of course. Running an experiment a second time to supply more data to further
confirm a hypothesis is still novel; after all, there’s now new data. Depending on various other factors, the
new data and what it implies, could be judged worthy of publication.
2
Some fields might dispute the value of real-world applicability in their research. I put this in, however,
because for ICTD research, I think it’s a common factor.
3
I’ve tried to put this very carefully, so as to encompass all the different kinds of generalizability. I might
have gone too far… One might ask, what isn’t generalizable according this definition? Maybe not much,
but the real question isn’t “generalizable or not?”, but “how generalizable?”.
variety of conditions; social theories are generalizable if they apply to different individuals or groups; many
of the tools of quantitative methods are meant to achieve generalizability; individual case studies, if framed
well, can add to a body of information from which generalizations can be extracted.4 In some cases,
generalizability is obvious (most ICTs work anywhere on the planet, within a reasonable range of climate),
and a write-up might tacitly assume it; in other cases, where it’s less clear, it’s good to outline the
boundaries of generalizability. Under what conditions do the conclusions apply, and why?

Rigor, verifiability, replicability: The research is done in a methodologically sound way so that it’s
possible to achieve and replicate the results as well as the analysis. Exactly what counts as
“methodologically sound” depends on the field from which the methodology was taken; most research
communities have good standards that are well-understood. Descriptions of the methodology used are
critical in papers. A useful question to ask is whether there is enough detail so that a competent graduate
student in the area could duplicate the research if she wanted to.

For any given research project, the above four elements might occur to varying degrees, but the more the
work has these qualities, the more likely it will be judged to be good research. Here are two examples of
good research, one from sociology and one from computer science: Everett Rogers’s sociological theory of
diffusion of innovations was (1) new at the time he synthesized it, (2) advisory, and therefore applicable,
with respect to how to spread innovations, (3) generalizable across different kinds of innovations and
human societies, and (4) based on rigorous research by many scholars, with clear circumscription of the
limits of the theory. In computer science, the transaction protocols for relational databases were (1) new at
the time they were developed, (2) applicable to a variety of information-storage and retrieval scenarios, (3)
generalizable to all kinds of data and systems, and (4) based on sound principles of logic with defined
conditions for how the protocols work. The respective research communities judged these contributions to
be significant because they demonstrated novelty, applicability, generalizability, rigor, all to great degrees.

Good research papers will highlight these characteristics in the writing. What is the novel contribution of
the research? What problem is being solved or what question is being answered by the work, and how does
the result support development? Under what conditions and to what extent does the work apply? What
methodology was followed in doing the research, and how could others with a basic background in the area
replicate or verify the results?

In contrast, a project that implements something already done before with the same results, has minimal
benefit, applies only to a particular small community, or was implemented with lax methodology, won’t be
considered good research.

Equally important is that papers be clear about the limitations of work along these axes. Although greater
novelty, applicability, generalizability, and rigor are desirable, it’s even more important to be accurate in
describing the extent of these qualities, so that readers can ascertain exactly how valid the conclusions are,
and where to begin if they want to extend the research.

Specifically ICTD
Relevance to ICTD: There is plenty of research in the world that is good research, but not ICTD research.
It should be obvious that ICTD involves both ICT and socio-economic development, and that each should
have an integral role in any work considered “ICTD.”5 The call for papers explains further.6

4
One of the values of theoretical frameworks in the social sciences is that they establish generalizable
hypotheses that new evidence can add to or detract from.
5
Every year the conference receives a few technical papers which are just about a new technology or
algorithm, and have made a thinly veiled attempt to cast the work as being applicable to development. Are
these people so desperate to present their work to an audience that doesn’t care? As a paper reviewer, it’s
easy to spot these papers and recommend for rejection.
6
http://ictd2009.org/cfp.html
ICTD is not the same as fields such as “Technology and Society”, or the “Information Society,” although
there is overlap. The focus, again, is very much on socio-economic development, and not just on the
relationship between technology and society in general.

If you believe your topic is concerned with development, but have doubts as to whether readers will agree,
it’s worth it to make your definition of development, as well as the potential development impact explicit.

Empiricism: Good ICTD research tends to be based on empirical evidence. Essays based solely on
speculation, philosophizing, or normative moralizing might be sought in other forums, but ICTD is inclined
toward evidence-based research. The evidence may be primary (you collect the evidence yourself) or
secondary (evidence collected or published by others), but conclusions should be based on real data and
systems should be tested in the anticipated environment.

If you did any kind of primary research or fieldwork in poor communities, I think it’s nice to supply a
concise explanation of the challenges in doing the work as part of the description of the methodology or the
discussion. Other disciplines often don’t recognize the value of these efforts (or at least, they don’t care to
hear about it), but in ICTD, experience in the field makes a huge difference. Your experiences could inform
future researchers, as well. As a side benefit, it also evokes empathy from reviewers, most of whom will
understand something of what you’ve gone through. (But, don’t overdo it, or it could have the opposite
effect.)

Audience Multidisciplinarity: There are anthropologists, sociologists, economists, engineers, computer

scientists, designers, etc., in the audience at ICTD, and the proceedings go online for an even wider
readership. So, when writing ICTD papers, it’s critical that most of the paper is comprehensible to a well-
educated lay reader, with minimal jargon and straightforward language. A good standard is English as it
appears in world-class newspapers and magazines, such as the New York Times, The Economist, or
Scientific American. Where jargon is necessary or helpful, it’s best to provide clear definitions. If very
specialized information is presented, it helps to interpret it in the writing for the intelligent layperson. The
ICTD review process involves at least three reviewers, of whom one have a background that is very
different from the home field of the paper, though they will be familiar with ICTD. 7 That person should
still be able to understand most of the paper.

Research Multidisciplinarity: Research that applies multiple disciplines is often what makes ICTD
different from its constituent disciplines (though this is not at all a requirement). The conference delights in
providing a home to good work that is hard to publish elsewhere. Some papers analyze a topic from
multiple perspectives, and offer a comprehensive analysis. The important thing here is not to compromise
on rigor and methodology, and to apply the right methodology for the right thing. Unless and until ICTD
develops its own unique methodologies (definitely a future possibility), it’s good to stick to established
methodologies from existing fields, since they have been tried, tested, and accepted by at least one research
community.

Other Key Elements

Related Work: Citing of previous, related work is a valuable academic tradition that exists in all
scholarship, and ICTD is no different. Because ICTD topics often touch on multiple disciplines, however,
it’s easy to overlook related work in fields you’re not familiar with. It’s worth the effort to find and cite
these, though, since there may be good lessons to learn, and bringing related work to the attention of the
ICTD community helps others learn what’s already been done, as well. (Not only that, some curmudgeonly
reviewer may not take it kindly that you’ve neglected to mention their entire field.) Another thing that can
help is to send drafts to colleagues who are not in your field for feedback. They may point you to related
work you weren’t aware of.

7
We do this so that authors receive feedback from people outside of their field. In the longer term, we hope
this will help establish a common vocabulary and writing culture within the ICTD community.
0% Hype: As researchers seeking recognition for our research, it’s only natural to want to show our work
in the best light. It’s my hope that in ICTD “the best light” is 0% hype and 100% reality. There is so much
hyperbole, exaggeration, and chest-beating in the development world at large, that it would be great if
ICTD became a haven of credibility. Analysis of intervention projects should contain discussions of their
limitations. Contrariwise, for those studies which end up being critiques, it would likewise be nice to situate
the critique in the larger context. An e-government project might indeed have failed to curb corruption, but
if it nevertheless provides an effective service and does so better than before, perhaps it could be a net
positive after all.8

Negative Results: Since the first call for papers in 2006, we wrote, “Well-presented negative results from
which generalizable conclusions can be drawn are also sought.” Negative results are when a project that
should have worked didn’t work, or when an expected trend didn’t materialize in the data. If the conditions
for the negative result are understood, generalizable, and previously unknown (i.e., not just another well-
known failure mode), it deserves to be heard. Articulating exactly what the new knowledge is, and why it’s
significant will be the challenge.

One Last Comment

If you’re new to ICTD, the best thing you can do is to read and deconstruct previous ICTD papers,
particularly those which were accepted as oral presentations. You can find these here:

ICTD2006 (Berkeley):
 Proceedings:
o http://ieeexplore.ieee.org (restricted access, unfortunately)
 Oral papers are listed in the program:
o http://www.ischool.berkeley.edu/ictd2006/pro.html
 The top papers from this conference were revised and published in Information Technology and
International Development (open access):
o http://www.mitpressjournals.org/toc/itid/4/1

ICTD2007 (Bangalore):
 Proceedings:
o http://research.microsoft.com/workshops/ictd2007/ICTD2007_Proceedings_CD.pdf
 Oral papers are listed in the program:
o http://research.microsoft.com/workshops/ictd2007/program.htm

Acknowledgements: Thanks to Jonathan Donner for feedback on an earlier draft.

8
Perhaps.