Critical appraisal for speech and language therapists: qualitative study

The purpose of this tool is to provide a structured framework for reading and appraising studies which use qualitative research methods. The strength of qualitative research lies in validity (closeness to the truth) that is, good qualitative research, using a selection of data collection methods, really should touch the core of what is going on rather than just skimming the surface. The validity of qualitative methods is greatly improved by the use of more than one method in combination (triangulation), by the researcher thinking carefully about what is going on and how their own perspective might be influencing the data ( reflexivity) (Greenhalgh, 2006, pp.168-9). Qualitative methods are the gold standard for exploring and creating meaning from participants subjective experiences, and for gaining insight into phenomena that are ill-defined or poorly understood. However, generalisation of findings to other contexts may not be justified without appeal to a wider evidence base. Useful short article: Kuper, A., Lingard, L. & Levinson, W. (2008) Critically appraising qualitative data, BMJ 337, pp.687-90 Use the following questions to assess the rigour, credibility and relevance of the study and whether there are implications for your service. There is space for you to add your own notes. The prompts are there to highlight why an issue is important but they shouldnt limit your thinking. There is quite a lot of overlap between questions.

Study reference:

Aims clearly stated?

Goal of the research?

Aims Why is it important?

How is it relevant?

Qualitative method appropriate? (Does the research seek to interpret or illuminate the actions and/or subjective experiences of research participants?

Study research design appropriate to address its aims? (Have they discussed how they decided which methods to use and justified their use?) Method

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Have they explained how the participants were selected?

Have they explained why the participants they selected were the most appropriate to provide access to the type of knowledge sought by the study? Sampling and recruitment

Were there are any discussions around recruitment (e.g. why some people chose not to take part)?

Was the setting for data collection justified?

Is it is clear how data were generated (focus group, semi-structured interview etc.)? Data generation Have they justified the methods chosen?

Were the methods made explicit (e.g. for interview method, is there an indication of how interviews were conducted, did they used a topic guide?)

Were methods modified during the study? If so, do they explain how and why?

Is the form of data clear (e.g. tape recordings, video material, notes etc)?

Did they discuss data saturation i.e. the point at which no new information is being generated?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Is there sufficient detail of how the research was explained to participants for the reader to assess whether ethical standards were maintained? (Bear in mind potential consequences when peoples personal experiences are exposed to public scrutiny.)

Ethical issues Have they discussed issues raised by the study (e. g. issues around informed consent or confidentiality or how they have handled the effects of the study on the participants during and after the study)?

Was approval gained from a recognised ethics committee?

Have the researchers critically examined their own role, potential bias and influence during formulation of research questions

Reflexivity
research partnership, relations, recognition of researcher bias

data generation, including sample recruitment and choice of location?

Do they explain how they responded to events during the study

considered the implications of any changes in the research design?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

NB Qualitative research often uses an iterative method where data generation and analysis proceed hand-in-hand. Each may influence the other in a cyclic fashion during the course of the study - in direct contrast to the consecutive data collection then analysis that is the standard practice in quantitative research. Is there an in-depth description of the analysis process?

Data analysis

Was thematic analysis used? If so, is it clear how the categories/themes were derived from the data?

Do they explain how the data presented were selected from the original sample to demonstrate the analysis process?

Are sufficient data presented to support the findings?

To what extent are contradictory data taken into account?

Did they critically examine their own role, potential bias and influence during analysis and selection of data for presentation?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Are the findings explicit?

Findings

Is there adequate discussion of the evidence both for and against the researchers arguments?

Do they discuss the credibility of their findings (e.g. triangulation, respondent validation, more than one analyst)? Do you believe them?

Are the findings discussed in relation to the original research questions?

Do they discuss the contribution the study makes to existing knowledge or understanding? (e.g. do they consider the findings in relation to current practice or policy, other relevant research-based literature or theory?)

Do they identify new areas where research is necessary? Value

Do they discuss whether or how the findings can be transferred to other populations or considered other ways the research may be used?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Do the study findings resonate with my own experiences (or those of my colleagues) and therefore have a potential impact for my practice

for my colleagues or care group

Impact Summary for the service as a whole?

Is there a further question to be asked?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme, and Tricia Greenhalgh (2006) How to read a paper, Oxford, Blackwell, by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2010) Journal club 2: qualitative research, Speech & Language Therapy in Practice Winter, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.