The use of medical information 1

The Use of Medical Information Retrieval Systems A Review of the Literature Christina Magnifico LI804XS

The use of medical information 2

Objectives: The literature review should help information professionals and researchers understand how to evaluate medical information retrieval systems, how medical information retrieval systems are currently utilized by clinicians, and identifies the search techniques used within medical information retrieval systems (MIRS) Method: A preliminary search of the literature on medical information retrieval systems in both PubMed and the LIS (library information science) databases returned articles on how information professionals are currently evaluating medical information retrieval systems, their general use within the clinical environment, and varied search techniques for returning relevant information.

Results: The results of this literature review show that medical information retrieval systems lack consistency and are not always efficient. Based on the way medical professionals use current medical information retrieval systems, consistency in how search queries are processed will require continued evaluation on the part of information professions and clinicians. Search Strategies: (Information Systems/utilization[MeSH Terms] AND ) Medical Staff, Hospital[MeSH Terms]; (Information Systems/utilization[MeSH Terms] AND ) librar* Key Concepts: medical information retrieval systems, information retrieval, medical librarianship, clinical librarianship, health science librarianship

Abstract: This paper contains a review of the medical and information science literature with a focus on the evaluation, usage, and search techniques within medical information retrieval systems. The review provides a detailed methodology of the search process used for the collection of the articles, so that the results can be replicated. This review shows that, although there are not currently set standards by which medical information retrieval systems are evaluated; the usage of electronic systems is on the rise, showing a need for query instruction. It also identifies several current trends within information retrieval system research and areas to be improved upon.

The use of medical information 3

The Use of Medical Information Retrieval Systems Information retrieval systems are an integral aspect of scientific research and have increasingly been used in clinical settings. Modern health care requires medical professionals to move beyond the hospital walls in order to incorporate evidence into standard patient care, thus increasing the need for digital information databases that can be accessed quickly and efficiently. The jump to evidence-based medicine now pushes researchers, clinicians, physicians and students to the limits of their searching abilities. Gone are the days of simply searching for citations and abstracts; now medical professionals must be able to synthesize the information they find into their daily routine. Rounds no longer take place around a table; instead mobile devices have allowed them to move into the clinical environment. With a vast world of information at their fingertips, the dissemination of medical information can now have an immediate and direct impact on the quality of care health care professionals can provide. This digital revolution within the health science community is making the timely retrieval of medical information of paramount import in the context of patient care. A review of the literature on the use of medical information systems reveals that research in this particular area follows three recurring themes: how to evaluate medical information retrieval systems; health care professionals use of medical information retrieval systems; and how to properly perform search queries within medical information retrieval systems. The research also shows that there is currently a lack of oversight in the evaluation of medical information retrieval systems. Since there is not a set of standards by which every medical information system are evaluated, this leads to inconsistency in both the way information retrieval systems are used by the medical community, as well as the process of performing search queries within individual databases.

The use of medical information 4

Methodology A preliminary search of the literature on medical information retrieval systems in both PubMed and the LIS databases returned articles on how information professionals are currently trying to evaluate existing medical information retrieval systems and their general use in the clinical environment. After briefly skimming the literature, I was able to use the cited references to find supplemental sources that bolstered my core articles. Many of the results returned were studies done by information professionals who were creating MIRS for use within their own academic institution. Several publications were literature reviews, while others were overviews of studies that fit within a certain theme. A fourth theme that came up within the search was the creation of custom MIRS, but I did not include these articles in my review of the literature because they did not provide information relevant to the use of MIRS in clinical settings. I used the following search strings in my search of both PubMed and the LIS databases through Emporia State: (Information Systems/utilization[MeSH Terms]) Medical Staff, Hospital[MeSH Terms]; (Information Systems/utilization[MeSH Terms]) librar* The first search I did was a broad search for medical information retrieval systems, which returned over 63,000 results. I quickly realized that this topic was much more diverse and complex than I had originally anticipated, so I went back to the drawing board and used the MeSH terms to help me narrow the search results. When I utilized the search strings mentioned above, I returned less than 300 results, which I further filtered using the Full text available filter in PubMed and the EBSCOhost (LIS) databases.

The use of medical information 5 Defining Information Retrieval Systems An information retrieval system is concerned with the storage, organization, and searching of collections of information (Larson, 2009). The term information retrieval, coined in 1950 by Calvin Mooers (Mooers, 1950), gained popularity with the advent of machine storage and electronic databases. Today, information retrieval systems are utilized in a variety of ways, though they are increasingly used to store and disseminate scientific and medical information. One of the first medical information retrieval systems, MEDLARS (MEDical Literature Analysis and Retrieval System) was shown to have a "58% recall and 50% precision" which identified a need for more thorough indexing and vocabulary development. Since MEDLARS, information technologists and clinicians have pioneered new medical information retrieval systems by working in tandem with the help of current research on the topic of medical information retrieval (Dee, 2007, p. 421). If medical information retrieval systems are to be of continued use, increased oversight and evaluation will be needed and many organizations, and individuals, are stepping up to the challenge. Evaluation of Medical Information Retrieval Systems Medical information retrieval systems are an important aspect of scientific research, but with new systems in development more frequently than before, each system must go through an evaluation process in order to make sure they provide both useful and accurate information to clinicians. In almost all cases, medical information retrieval systems must adhere to the strict National Library of Medicine (NLM) criteria for performance, currency, and updating (Haynes, Walker, McKibbon, Johnston, & Willan, 1994). Even though the NLM has established basic criteria, it is important that medical information retrieval systems go through the evaluation process in a number of ways. One of the main reasons for the evaluation of medical information

The use of medical information 6 retrieval systems is that the overall complexity of health care information, systems constraints and user and institution needs are continually changing (Deibel & Greenes, 1995, p. 765). Several of the articles reviewed (Demiris, Folk, Mitchell, Moxley, Patrick & Tao, 2003 Grandage, Slawson & Shaughnessy, 2002 Liu & Wyatt 2011) evaluated the literature based on the type of study returned, with a select few focusing on with systematic reviews and metaanalyses (Demiris, Folk, Mitchell, Moxley, Patrick & Tao, 2003) or assessing the number of randomized-controlled trials returned (Liu and Wyatt, 2011).

The use of medical information 7 One image that represents the results of studies focused on the type of articles reviewed and returned is the hierarchy of studies (Figure 1). This figure shows that Cochrane Reviews, which contain systematic reviews as well as meta-analyses, are highly prized in the academic community.

Figure 1 Hierarchy of Studies

Focusing solely on the number of meta-analysis returned without extensive knowledge of research methodology, can be problematic, however. Demiris (2003), conducted a study using Ovid MEDLINE, the articles returned were evaluated by dividing them into subsets based on the

The use of medical information 8 level of effectiveness seen in their retrieval strategies. To evaluate a medical information retrieval system based on a singular type of study is both shortsighted and time-consuming. Circumventing the process of whittling away at meta-analyses or randomized controlled trials, Grandage, Slawson and Shaughnessy (2002) used a simple equation in order to evaluate the usefulness of articles returned from a medical information retrieval system:

This equation, which has the potential to mine information from multiple medical information retrieval systems simultaneously, saves both the researcher and user time when querying medical information retrieval systems. Mining, mentioned previously, is a valuable data resource (Del Fiol & Haug, 2008), as it can collect usage statistics without the need to be constantly monitored by a human being. When evaluating an information retrieval system, Grandage (2002) places increased emphasis on the role medical librarians should take in order to provide the most assistance to clinicians. Other articles (Elliot, Hersh, Hickam & Wolf, 1994 Masic & Milinovic, 2012 Braithwaite, Coiera & Westbrook, 2005) focus primarily on evaluating the entire medical information retrieval system as a whole. Cimino, Ely, Lee, Sable, Shanker and Zhu (2006) evaluated medical information retrieval systems based on the number of documents returned after using a question answering technique, while a 2006 study by MacCall focused on assessing information retrieval systems and clinical digital libraries in terms of their facilitation of timely clinical information seeking. However, medical information retrieval systems contain more than just text-based documents. Medical images are an integral part of clinical diagnosis, analyzing treatment options and evaluating laboratory results (Demner-Fushman, 2009). Searching for medical

The use of medical information 9 images within medical information retrieval systems, however, is different than searching for articles. As stated in Demner-Fushman (2009), "Currently available information retrieval and decision support systems rely primarily on the text of scientific publications to find evidence in support of clinical information needs (p. 1)." Without consistent evaluations of tagging within medical information retrieval systems, searching for images in a text-based system will be a frustrating experience. Many medical images are out of context and a simple caption, although searchable, does not provide the clinician with relevant information about the image (p. 2). Timely, well-categorized images are important to clinical diagnosis, especially in certain specialties like dermatology. Searching for those images can be inefficient due to the current state of results returned. Based on the findings in Demner-Fushman (2009), images clearly benefited when combined with textual features, especially clinically relevant images. However, the overall quality of image retrieval has room to improve in the area of result reliability (pp., 89). Throughout the literature, several common ways to evaluate medical information retrieval systems, both text-based and image-based appeared. The most common include measuring usage frequency and measuring users' success at retrieving relevant information (Hersh et al., 1994, p. 895). Another way to evaluate medical information retrieval systems is by using recall-precision analysis, which is done by analyzing search logs (Hersh, 2002, p. 287) and evaluating the results returned. Other ways of evaluating medical information retrieval systems include frequency of use, purpose of use, user satisfaction, search failure, and searching utility, which includes answering clinical questions, retrieval of relevant articles and relevancebased measures in bibliographic systems (Hersh & Hickam, 1998). All of these ways of evaluating medical information retrieval systems assist in making them usable by health care

The use of medical information 10 professionals, though continuing evaluation of all systems is a requirement for continued success. Health Professionals Use of Medical Information Retrieval Systems One of the most frequent themes in the review of the literature was how clinicians, researchers and physicians use medical information retrieval systems. Many times, however, physicians find the process of looking for articles in a database time-consuming and difficult. They often run into obstacles that cause frustrations, which in turn push them away from using certain electronic resources and databases (Fauqert, 2012, p. 401). A multitude of studies (Cimino, Hunt & Koziol, 2012 Clevenger & Shelstad, 1996 Coiera, Fauquert, 2012, Gosling & Westbrook, 2005 Grad, Meng, Pluye, Segal & Tamblyn, 2005 Grefsheim & Rankin, 2007 Haynes, 1994 Hersh & Hickam, 1998) focused on IRS usage by the medical community. A study conducted by Haynes (1994) showed that clinicians use of MEDLINE had risen during the past five years. This increase in use came from a variety of reasons including, "user-friendly software, a proliferation of online and compact-disc formats, falling user charges, and advertising directed at clinicians." Though each of the studies sought to identify the way different medical professionals were utilizing medical information retrieval systems, their methodologies were not always the same. The environment in which the systems were utilized was an important factor in choosing the methodology by which to study clinicians use (Cimino, 2012). Whether it be by analyzing retrieval patterns among general surgeons (Clevenger, 1996) or family practitioners (Grad, 2005 Grefsheim, 2007 Hersh, 1998), evaluating the effects of the dissemination of information by physicians (Fauquert, 2012), or comparing the use of pre-versus post-intervention question

The use of medical information 11 answering by clinicians (Coiera, 2005), each identified the environment as important. One of the most important uses of medical information retrieval systems was in the realm of public health. Many public health professionals need relevant information quickly and easily. Alpi (2005) stated, "Knowing how to quickly locate materials in multidisciplinary full-text databases is important to public health searching." However, due to the wide variety of databases used in medicine, especially in regards to public health, efficient searching is key to the success of clinicians (Alpi, 2005, p. 4). When faced with literature that is difficult to find, searching multiple databases, each with its own limitations, is time consuming and inefficient. The amount of Websites and databases available to medical professionals is increasing exponentially. With growth, however, come consequences; mainly in that a consistent level of expertise is difficult to maintain with an ever-changing search landscape (Alpi, 2005, p. 5). As the practice of evidence-based medicine continues to increase, more clinicians will begin to integrate their clinical expertise with the most relevant clinical evidence culled from medical databases (Chung, 2009, p. 1). It was noted in a 2005 study by Grad that "the tremendous volume of clinical information makes it difficult for doctors to rapidly access what they need (p. 582)." They also stated that, "an information need is recognized when a doctor reflects on their practice, and asks a question. For example, does the patient have acute sinusitis? The framework proposes that doctors may recognize their information needs, pursue and satisfy them, and subsequently implement information in their decision making (as cited in Ebell, 2003, p. S53). It is important to identify the needs of physicians and clinicians, as the information needs of practicing clinicians are distinct from the needs of students, researchers, or nonclinical personnel. Clinicians seek information to stay current with new relevant medical developments and to find answers to patient-specific questions (Grandage, 2002, p. 298)."

The use of medical information 12 "Clinicians [also] generate highly specific patient-related questions at a rate of about one to three questions for every three patient visits. Of every ten questions posed, they only look up the answers to four and only find the answers to three. Of those they do not look up, they estimate [that] at least half are important. Thus, clinicians are guessing at seven of ten questions, due in large part to the amount of work it takes to find valid and reliable information that applies to their patients (Grandage, 2002, p. 299). Help is on the way, however. Darmoni (2012) posited that, "it can be expected that users search experiences in MEDLINE [as well as other medical information retrieval systems] will be enhanced by techniques whereby both database and search engine developers make full use of the MeSH structure" (p. 181). Regular search engines do not have the complexity to return relevant results when queried with complex scientific requests, thus their use by clinicians falls short in the area of scientific literature searching. Unlike computer-based medical information retrieval systems, regular search engines cannot filter relevant results efficiently, making the extraction process frustrating and difficult (Earl, 2010, p. 1). In order to begin assessing the search situation, mining studies have been conducted. A French study of Entrz, an NCBI integrated search system that handles more than 3 million searches daily, [showed that] 70% of the searches are done as simple queries, in which terms are entered without field specifiers, Boolean queries, or other advanced search techniques that can be applied to achieve more meaningful search results. Just 21% of the searches use Boolean operators, 13% use field specifiers, and 1% each use wildcard, range searching, and the History function (Geer, 2006, p. 290)." Without the proper training, health care professionals use of medical information retrieval systems falls short of their initial expectations. This could cause a

The use of medical information 13 decline in the use of medical information retrieval systems by the scientific community as whole, if instructional measures are not put into place. Performing Search Queries in Medical Information Retrieval Systems Many clinicians feel that they are adequately able to search within medical information retrieval systems. Haynes (1994) demonstrated that clinicians who had experience with database searching retrieved relevant citations, but their citations were not nearly as precise as librarians were. However, with search training, clinicians "acquired comparable proficiency to that of librarians [after several searches] (p. 293)." By using controlled vocabularies or Medical Subject Headings (MeSH Terms), clinicians are able to replicate searches performs by information professionals. However, Alpi (2005) discussed the shortfalls of controlled vocabularies, particularly Medical Subject Headings (MeSH): Controlled vocabularies in health databases do not capture the importance of place and partners well. For example, in Medical Subject Headings (MeSH), the terms neighborhood, community, place of birth, living arrangements, and domicile are all entry terms mapping to the MeSH term Residence Characteristics, which is defined as Elements of residence that characterize a population. (p. 2). Since MeSH Terms are not consistent between all medical information retrieval systems, many physicians tend to stick with a single system instead of exploring other options. In order to assist physicians in their searching, one study (Bekhuis, Demner-Fushman and Crowley, 2013) provided an extensive crosswalk of MeSH headings and Emtree matches (Table 1) that could be of use to librarians and researchers. The crosswalk identified major differences between the two databases, which could be cause for confusion and frustration to novice searchers. However, consistency in search terminology is not the only shortfall within medical information retrieval systems.

The use of medical information 14 There have been efforts to create a more user-friendly search interface within MIRS. In PubMed a study was undertaken to address the way rankings are evaluated and how query strategies affect the PubMed search engine (Darmoni et al., 2012). In order to give a better overview of the current set-up of many medical information retrieval systems, particularly the Unified Medical Language System (UMLS) used within them, Darmoni (2012) described the 2011 version of the MeSH Thesaurus, which "contains 26,142 Descriptors, 83 Qualifiers, 25,801 Entry Terms, 200,676 Supplementary Concepts, and 317,554 Concepts." A structure this complex can make searching the system an extremely convoluted and inefficient process. The use of filters can help alleviate some of the burden of search inefficiencies. In evaluating the literature, many of the articles (Abernethy, Currow, Fazekas, Sladek & Tieman 2006 Carlson, Cvitan, Krieger, Lavin, McNary, Meyer, Perry, Reese & Spasser, 2005 Meyer & Sarin, 2005 Stoddart & Workman, 2012) described studies on the usage of filters in medical information retrieval system searching. Search filters are developed to improve the efficiency and effectiveness of searching and are typically created by identifying and combining search terms to retrieve records with a common feature (Glanville, 2008, p. 356). Each article that focused on search filters, used a different set of filters: palliative care search filters (Abernathy, 2006), evidence-based search filters (Carlson, 2005), anesthesia-related search filters (Meyer & Sarin, 2005), and natural language filters (Stoddart & Workman, 2012). One set of filters, a semi-manually constructed Boolean [query] of MeSH terms, publication type and MEDLINE record, which was designed by Haynes (1994) are the most widely available method for identifying high quality articles through PubMed (Fu, 2011). The ability to perform efficient, relevant searches within medical information retrieval systems is important to clinical practices and evidence based medicine. "In 1996, the National Library of

The use of medical information 15 Medicine addressed the need for clinicians to refine their MEDLINE search retrieval in PubMed by applying proven clinical filters. Clinical Queries provide a way to limit search retrieval to articles about the four types of clinical research: diagnosis, etiology, therapy, and prognosis, as well as options to direct the emphasis of the search to be more sensitive or more specific (Grandage, 2002, p. 300). Though there have been many studies which have taken up the task of improving the search function within medical information retieveal systems, few have gone so far as to implement the results. One suggestion by Haynes (1994) was to develop a potential method for improving the detection of studies of high quality for clinical practice from MEDLINE. A way to implement this is to include search terms that select studies that are at the most advanced stages of testing for clinical application (p.457). As advances in natural language processing, machine learning, and metadata tagging continue to improve, the way clinicians search for information within medical information retrieval systems will surely evolve. Conclusion As medical information retrieval systems become increasingly prevalent within clinical practice, improved oversight and better evaluation is needed. A definitive set of standards by which to judge new and existing systems needs to be set in place, especially as access to medical information retrieval systems increases. Studies into the use of medical information retrieval systems will continue to be integral to the evaluation of the systems as a whole. Since user experience is such an important aspect of designing information retrieval systems, the feedback from the medical community and the data mined from their usage will be vital to those tasked with designing information retrieval systems for the next generation of users. Not only does this include superior interface design, but enhanced search processing and more efficient use of existing search processing.

The use of medical information 16 References Alpi, K. M. (2005). Expert searching in public health. Journal of the Medical Library Association, 93(1), 97103. Bekhuis, T., Demner-Fushman, D., & Crowley, R. S. (2013). Comparative effectiveness research designs: an analysis of terms and coverage in Medical Subject Headings (MeSH) and Emtree. Journal of the Medical Library Association: JMLA, 101(2), 92100. doi:10.3163/15365050.101.2.004 Chung, G. Y. (2009). Sentence retrieval for abstracts of randomized controlled trials. BMC Medical Informatics and Decision Making, 9, 10. doi:10.1186/1472-6947-9-10 Darmoni, S. J., Soualmia, L. F., Letord, C., Jaulent, M.-C., Griffon, N., Thirion, B., & Neveol, A. (2012). Improving information retrieval using Medical Subject Headings Concepts: a test case on rare and chronic diseases. Journal of the Medical Library Association: JMLA, 100(3), 176183. doi:10.3163/1536-5050.100.3.007 Dee, C. R. (2007). The development of the Medical Literature Analysis and Retrieval System (MEDLARS). Journal of the Medical Library Association: JMLA, 95(4), 416425. doi:10.3163/1536-5050.95.4.416 Deibel, S. R. A., & Greenes, R. A. (1995). An infrastructure for the development of health care information systems from distributed components. Journal of the American Society for Information Science, 46(10), 765771. doi:10.1002/(SICI)1097-4571(199512)46:10<765::AIDASI8>3.0.CO;2-W Del Fiol, G., & Haug, P. J. (2008). Infobuttons and classification models: a method for the automatic selection of on-line information resources to fulfill clinicians information needs. Journal of biomedical informatics, 41(4), 655666. doi:10.1016/j.jbi.2007.11.007

The use of medical information 17 Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R. (2009). Annotation and retrieval of clinically relevant images. International journal of medical informatics, 78(12), e59e67. doi:10.1016/j.ijmedinf.2009.05.003 Earl, Martha F. (2010, April). Information Retrieval in Biomedicine: Natural Language Processing for Knowledge Integration. NCBI. Database. Retrieved May 20, 2013, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2859258/ Fauquert, B. (2012). [From library to clinical decision support systems: access of general practitioner to quality information]. Revue mdicale de Bruxelles, 33(4), 400406. Fu, L. D., Aphinyanaphongs, Y., Wang, L., & Aliferis, C. F. (2011). A Comparison of Evaluation Metrics for Biomedical Journals, Articles, and Websites in Terms of Sensitivity to Topic. Journal of biomedical informatics, 44(4), 587594. doi:10.1016/j.jbi.2011.03.006 Gault, L. V., Shultz, M., & Davies, K. J. (2002). Variations in Medical Subject Headings (MeSH) mapping: from the natural language of patron terms to the controlled vocabulary of mapped lists. Journal of the Medical Library Association, 90(2), 173180. Geer, R. C. (2006). Broad issues to consider for library involvement in bioinformatics. Journal of the Medical Library Association, 94(3), 286E155. Glanville, J., Bayliss, S., Booth, A., Dundar, Y., Fernandes, H., Fleeman, N. D., Welch, K. (2008). So many filters, so little time: the development of a search filter appraisal checklist. Journal of the Medical Library Association: JMLA, 96(4), 356361. doi:10.3163/1536-5050.96.4.011 Grad, R. M., Pluye, P., Meng, Y., Segal, B., & Tamblyn, R. (2005). Assessing the impact of clinical information-retrieval technology in a family practice residency. Journal of Evaluation in Clinical Practice, 11(6), 576586. doi:10.1111/j.1365-2753.2005.00594.x

The use of medical information 18 Grandage, K. K., Slawson, D. C., & Shaughnessy, A. F. (2002). When less is more: a practical approach to searching for evidence-based answers. Journal of the Medical Library Association, 90(3), 298304. Grefsheim, S. F., & Rankin, J. A. (2007). Information needs and information seeking in a biomedical research setting: a study of scientists and science administrators. Journal of the Medical Library Association: JMLA, 95(4), 426434. doi:10.3163/1536-5050.95.4.426 Haynes, R. B., Wilczynski, N., McKibbon, K. A., Walker, C. J., & Sinclair, J. C. (1994). Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association: JAMIA, 1(6), 447458. Hersh, W. R., Elliot, D. L., Hickam, D. H., Wolf, S. L., Molnar, A., & Leichtenstien, C. (1994). Towards new measures of information retrieval evaluation. Proceedings of the Annual Symposium on Computer Application in Medical Care, 895899. Hersh, William R., Crabtree, M. K., Hickam, D. H., Sacherek, L., Friedman, C. P., Tidmarsh, P., Kraemer, D. (2002). Factors Associated with Success in Searching medline and Applying Evidence to Answer Clinical Questions. Journal of the American Medical Informatics Association: JAMIA, 9(3), 283293. doi:10.1197/jamia.M0996 Hersh WR, & Hickam DH. (1998). How well do physicians use electronic information retrieval systems?: A framework for investigation and systematic review. JAMA, 280(15), 13471352. doi:10.1001/jama.280.15.1347 Hunt, S., Cimino, J. J., & Koziol, D. E. (2013). A comparison of clinicians access to online knowledge resources using two types of information retrieval applications in an academic hospital setting. Journal of the Medical Library Association: JMLA, 101(1), 2631. doi:10.3163/1536-5050.101.1.005

The use of medical information 19 Hwang, K. H., Lee, H., & Choi, D. (2012). Medical Image Retrieval: Past and Present. Healthcare Informatics Research, 18(1), 39. doi:10.4258/hir.2012.18.1.3 Information Retrieval. (n.d.). OpenClinical.org. Retrieved May 20, 2013, from http://www.openclinical.org/informationretrieval.html Jenuwine, E. S., & Floyd, J. A. (2004). Comparison of Medical Subject Headings and text-word searches in MEDLINE to retrieve studies on sleep in healthy individuals. Journal of the Medical Library Association, 92(3), 349354. Karimi, S., Pohl, S., Scholer, F., Cavedon, L., & Zobel, J. (2010). Boolean versus ranked querying for biomedical systematic reviews. BMC Medical Informatics and Decision Making, 10, 58. doi:10.1186/1472-6947-10-58 Kingsley, K., Galbraith, G. M., Herring, M., Stowers, E., Stewart, T., & Kingsley, K. V. (2011). Why not just Google it? An assessment of information literacy skills in a biomedical science curriculum. BMC Medical Education, 11, 17. doi:10.1186/1472-6920-11-17 Lavin, M. A., Krieger, M. M., Meyer, G. A., Spasser, M. A., Cvitan, T., Reese, C. G., McNary, P. (2005). Development and evaluation of evidence-based nursing (EBN) filters and related databases. Journal of the Medical Library Association, 93(1), 104115. Lee, M., Cimino, J., Zhu, H. R., Sable, C., Shanker, V., Ely, J., & Yu, H. (2006). Beyond Information Retrieval--Medical Question Answering. AMIA Annual Symposium Proceedings, 2006, 469473. MacCall, S. L. (2006). Clinical Digital Libraries Project: design approach and exploratory assessment of timely use in clinical environments. Journal of the Medical Library Association, 94(2), 190 197.

The use of medical information 20 Masic, I., & Milinovic, K. (2012). On-line Biomedical Databases- The Best Source for Quick Search of the Scientific Information in the Biomedicine. Acta Informatica Medica, 20(2), 7284. doi:10.5455/aim.2012.20.72-84 Mosa, A. S. M., & Yoo, I. (2013). A Study on Pubmed Search Tag Usage Pattern: Association Rule Mining of a Full-day Pubmed Query Log. BMC Medical Informatics and Decision Making, 13, 8. doi:10.1186/1472-6947-13-8 Mu, X., Lu, K., & Ryu, H. (2010). Search strategies on a new health information retrieval system. Online Information Review, 34(3), 440456. doi:http://dx.doi.org/10.1108/14684521011054062 Nicholson, S. (2005). Understanding the foundation: the state of generalist search education in library schools as related to the needs of expert searchers in medical libraries. Journal of the Medical Library Association, 93(1), 6168. Patrick, T. B., Demiris, G., Folk, L. C., Moxley, D. E., Mitchell, J. A., & Tao, D. (2004). Evidencebased retrieval in evidence-based medicine. Journal of the Medical Library Association, 92(2), 196199. Performances of 27 MEDLINE systems tested by searches with clinical questions. (n.d.). Retrieved May 20, 2013, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC116206/?tool=pubmed Pletneva, N., Vargas, A., Kalogianni, K., & Boyer, C. (2012). Online health information search: what struggles and empowers the users? Results of an online survey. Studies in health technology and informatics, 180, 843847. Shelstad, K. R., & Clevenger, F. W. (1996). Information retrieval patterns and needs among practicing general surgeons: a statewide experience. Bulletin of the Medical Library Association, 84(4), 490497.

The use of medical information 21 Shultz, M. (2006). Mapping of medical acronyms and initialisms to Medical Subject Headings (MeSH) across selected systems. Journal of the Medical Library Association, 94(4), 410414. Sladek, R., Tieman, J., Fazekas, B. S., Abernethy, A. P., & Currow, D. C. (2006). Development of a subject search filter to find information relevant to palliative care in the general medical literature. Journal of the Medical Library Association, 94(4), 394401. Snapshot. (n.d.). Retrieved from http://search.proquest.com/docview/578105319?accountid=28920 Soualmia, L. F., Prieur-Gaston, E., Moalla, Z., Lecroq, T., & Darmoni, S. J. (2012). Matching health information seekers queries to medical terms. BMC Bioinformatics, 13(Suppl 14), S11. doi:10.1186/1471-2105-13-S14-S11 Watson, L. A. (2005). Information Retrieval: A Health and Biomedical Perspective. Journal of the Medical Library Association, 93(1), 134135. Westbrook, J. I., Coiera, E. W., & Braithwaite, J. (2005). Measuring the Impact of Online Evidence Retrieval Systems using Critical Incidents & Journey Mapping. Studies in health technology and informatics, 116, 533538. Westbrook, J. I., Coiera, E. W., & Gosling, A. S. (2005). Do Online Information Retrieval Systems Help Experienced Clinicians Answer Clinical Questions? Journal of the American Medical Informatics Association: JAMIA, 12(3), 315321. doi:10.1197/jamia.M1717 Whipple, E. C., McGowan, J. J., Dixon, B. E., & Zafar, A. (2009). The selection of high-impact health informatics literature: a comparison of results between the content expert and the expert searcher. Journal of the Medical Library Association: JMLA, 97(3), 212218. doi:10.3163/1536-5050.97.3.010 Yu, H., & Cao, Y. (2009). Using the Weighted Keyword Model to Improve Information Retrieval for Answering Biomedical Questions. Summit on Translational Bioinformatics, 2009, 143147.

The use of medical information 22

Appendix

Table 1. An example of the spreadsheet in which the authors compiled the crosswalk