Research Poster

The Research Paper Review Process and Advice for New Authors and Reviewers

Alyssa Rockers
Oklahoma State University
405-744-9464
alyssa.rockers@okstate.edu

Quisto Settle
Oklahoma State University
405-744-6548
qsettle@okstate.edu
 Research Poster

Introduction & Theoretical Framework

The purpose of this study was to identify the processes that reviewers for agricultural
communications journals use when reviewing pieces for publication. In addition, it served to
collect advice new reviewers and new authors. No discipline has found an ideal review process,
making it necessary for continuing research on this topic in all disciplines (Shoulders, 2015),
including agricultural communications. Guidelines for reviewers vary, and many times they are
passed on through formal or informal mentorship (Adamson, 2012). In addition, while some
journals provide editorials or other means to assist their reviewers in the reviewing process
(Eynon, 2014), bias in research publication still exists within disciplines (Carroll, 2017). The
Journal of Applied Communications utilizes guidelines for reviewers that were adapted from
Leila Jones’ 2017 recommendations in the Taylor and Francis Group’s editor resources (Jones,
2017). These guidelines, however do not describe the processes that many reviewers use when
reviewing articles for their selected journals.

Research and editorials in other disciplines have been published to try to standardize and gain
insight into the peer review process. Kachelmeier (2004), in his observations following the
American Taxation Panel, shared his advice for reviewers: summarize (contribution then
assessment), consider optimal length, be aware of bias, and be tactful. His final point of tact is
noted due to the lack of humanization that occurs when reading a manuscript (Kachelmeier,
2004). What is not noted in Kachelmeier’s report is the struggle that many journals have to find
those willing reviewers. Tite and Schroter (2007) found that many potential reviewers were
overworked and undermotivated to provide reviews to academic journals. Many who did choose
to review did so because they felt a topic was very relevant to the subject or their own work
(Tite, 2007). Adamson (2012) recommended that journals implement mentorship programs to
assist new reviewers, as many young reviewers were already seeking mentors. Adamson found
that not only did these mentorship programs assist new reviewers, they also helped more senior
editors and reviewers have a heightened purpose (Adamson, 2012). In an editorial to Learning,
Media and Technology, Enyon (2014) encouraged reviewers to keep their reviews between 300-
500 words and to think about their audiences, be specific, be strict but fair, be timely, and
upfront. In terms of advice to authors, Lee (2011) found that editors of health and toxicology
journals “frequently have to make a judgement call about the potential impact of submitted
papers” (p. 898). The editors shared that many times papers fit their journal criteria, yet do not
contain new information (Lee, 2011). Kachelmeier (2004) provided advice for authors in
addition to his advice to reviewers. To authors of research, Kachelmeier advised to beware of
errors of substance and communication, that the response memorandum is not the best way to
respond to reviews, to write to communicate, and to keep the faith (2004). In his note on keeping
the faith, Kachelmeier emphasizes that reviewers often disagree and that “empirical studies of
actual reviewer ratings reveal correlations that are embarrassingly low” (p. 152, 2004).

Methods

A list of 43 reviewers in agricultural communications was sent an anonymous link to participate

in the questionnaire, and 22 (51.2%) completed the qualitative questionnaire. To help increase
response rate, an initial email invitation was sent, along with two reminder emails over the
course of 12 days. No further reminders were sent once the number of new responses to
 Research Poster

reminders dwindled enough to no longer justify additional reminders (Dillman, Smyth, &
Christian, 2014). Participants were asked how much time they allot for reviewing, what their
process for reviewing, their advice for new reviewers, and their advice for new authors writing
for the peer-review process. These questions were part of a larger instrument assessing what
reviewers were looking for as they reviewed research papers. The participants consisted of eight
assistant professors, seven associate professors, and six professors. The results were analyzed for
categories of responses for the open-ended items.

Results

The participants reported time spent reviewing that ranged from 30 minutes to eight hours. Of
the 20 participants, nine reported spending 30 minutes to an hour reviewing, while six reported
spending two to three hours. In addition, two participants reported spending four hours, and one
reported spending six to eight hours. The participants had a lot of variety in their reported
process for reviewing, although seven participants reported utilizing the journal’s rubric or
guidelines in their process. Two reviewers stated that they print papers to review them, while one
stated that they prefer to use the track changes function in Microsoft Word. For advice for new
reviewers, seven participants included a version of “take your time”, although other advice
varied. Four participants advised new reviewers to be critical and constructive with their
feedback. Advice to authors had much less consistency, although seven of the reviewers advised
new authors to spend a substantial amount of time reading previous work published in journals
where they hope to be published.

Conclusions & Recommendations

The peer review process determines high-quality academic publishing, although it can often be
confusing for new reviewers (Eynon, 2014). The results of this study related to reviewing
processes can help new reviewers in the agricultural communications discipline develop their
own processes by helping them understand norms, which can hopefully produce high quality
reviews earlier in their career. Authors in turn can utilize these findings to maximize their
chances of being published. In addition, the time spent reviewing in the agricultural
communications discipline is valuable in order to understand how much time reviewing takes
from a reviewer’s other work (Tite, 2007). Ferris (2010) recommended that institutions use
review contributions to determine things such as tenure, salary, and academic rank. However,
because time spent reviewing varies so much (between 30 minutes and 8 hours according to
participants), it would difficult for universities to accurately estimate how much time each person
spends reviewing based solely on the number of reviews they have completed.

Future research should assess the viability of mentorship for helping to improve the research
review process, including any mentorship current faculty received, including formally or
informally. In addition, research should identify if there is a perceived bias in the types of articles
being published in the discipline, such as qualitative or quantitative, or between different
theoretical areas in agricultural communications. Finally, research should identify what value, if
any, authors are finding in the review process. Ultimately, reviews should improve the paper, but
if authors are not valuing the process, the discipline can assess how to make the review process
more productive.
 Research Poster

References
Adamson, J. (2012). Mentoring academic journal reviewers: brokering reviewing
knowledge. Innovations in Education & Teaching International, 49(2), 223–232.
doi:10.1080/14703297.2012.677593
Carroll, H. A., Toumpakari, Z., Johnson, L., & Betts, J. A. (2017). The perceived feasibility of
methods to reduce publication bias. PLoS ONE, 12(10), 1–19.
doi:10.1371/journal.pone.0186472
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode
surveys: The tailored design method (4th ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Eynon, R. (2014). How to review a journal article: Questions of quality, contribution, and
appeal. Learning, Media and Technology, 39(2), 151–153.
doi:10.1080/17439884.2014.888354
Ferris, L. E., & Brumback, R. A. (2010). Academic merit, promotion, and journal peer
reviewing: The role of academic institutions in providing proper recognition. Journal of
Child Neurology, 25(5), 538–540. doi:10.1177/0883073810365011
Jones, L. (2017, June 22). Reviewer guidelines and best practice. Taylor and Francis Group.
Retrieved from: http://editorresources.taylorandfrancisgroup.com/reviewers-guidelines-
and-best-practice/
Kachelmeier, S. J. (2004). Reviewing the Review Process. Journal of the American Taxation
Association, 26, 143–154.
Lee, B. M. (2011). Evaluation Criteria for Publishing in Top-Tier Journals in Environmental
Health Sciences and Toxicology. Environmental Health Perspectives, 119(7), 896–899.
doi:10.1289/ehp.1003280
Journal of Applied Communications. (n.d.) Policies. Available at
https://newprairiepress.org/jac/policies.html#reviewers
Shoulders, C. W., Johnson, D. M., & Flowers, J. (2015). Peer Review in Agricultural Education:
Interrater Reliability of Manuscript Reviews for the 2014 National Agricultural
Education Research Conference. Journal of Agricultural Education, 56(3), 121-133.
doi:10.5032/jae.2015.03121
Tite, L., & Schroter, S. (2007). Why do peer reviewers decline to review? A survey. Journal of
Epidemiology & Community Health, 61(1), 9–12. doi:10.1136/jech.2006.049817