Last week’s blog focused on the strongest types of evidence that you might find when trying to solve a clinical problem. These are: #1 Systematic reviews, Meta-analyses, or Evidence-based clinical practice guidelines based on systematic review of RCTs; & #2 Randomized controlled trials. (For levels of evidence from strongest to weakest, see blog “I like my coffee (and my evidence) strong!”)
So after the two strongest levels of evidence what is the next strongest? #3 level is controlled trials without randomization. (Sometimes called quasi-experimental studies.)
Here’s an example of a controlled trial without randomization: I take two groups of mice and test two types of cheese to find out which one mice like best. I do NOT randomly assign the mice to groups. The experimental group #1 loved Swiss cheese, & the control group #2 refused to eat the cheddar. I assume confidently that mice LOVE Swiss cheese…
Are we talking cigarettes? water? coffee? other? Yes, other. In this case about what is sometimes called“filtered” or “unfiltered” literature in the evidence-based medicinepyramid of research evidence. (I have more than one issue with this particular pyramid as a representation of all evidence, but for right now let’s look at filtered information & unfiltered information. Pyramid source: Wikimedia Commons
Filtered is considered stronger–meaning that we can be more confident that literature from this category better supports cause and effect. I agree.
Unfiltered evidence (usually single studies etc) is considered weaker–meaning that we must be more cautious about its accuracy in representing reality. I agree.
But, “Is unfiltered information really unfiltered?” No filtering at all? My qualified answer is, “No.” Argue with me if you like.
My opinion: If the “unfiltered” article is a primary source, research study that has strong design and is published in a peer-review journal then it has been filtered by multiple, expert peer reviewers just to make it to publication.
Thus, when discussing filtered vs. unfiltered one should be very clear on what those terms mean and do not mean.
Critical Thinking: When filtered literature (systematic reviews & critically appraised topics & articles) has been filtered by one individual, is that superior to unfiltered literature in terms of introducing bias? What if the “filtered” evidence is 7 years old and a primary, “unfiltered” source(s) from this year has different findings? What is the relationship between “filtered” and “unfiltered”–after all the “unfiltered” is the pyramid base so what does that mean?
Self-report by participants is one of the most common ways that researchers collect data, yet it is fraught with problems. Some worries for researchers are: “Will participants be honest or will they say what they think I want to hear?” “Will they understand the questions correctly?” “Will those who respond (as opposed to those who don’t respond) have unique ways of thinking so that my respondents do not represent everyone well?” and a BIG worry “Will they even fill out and return the questionnaire?”
One way to solve at least the latter 2 problems is to increase the response rate, and Edwards et al (2009 July 8) reviewed randomized trials to learn how to do just that!!
If you want to improve your questionnaire response rates, check it out! Here is Edwards et al.’s plain language summary as published in Cochrane Database of Systematic Reviews,where you can read the entire report.
Postal and electronic questionnaires are a relatively inexpensive way to collect information from people for research purposes. If people do not reply (so called ‘non-responders’), the research results will tend to be less accurate. This systematic review found several ways to increase response. People can be contacted before they are sent a postal questionnaire. Postal questionnaires can be sent by first class post or recorded delivery, and a stamped-return envelope can be provided. Questionnaires, letters and e-mails can be made more personal, and preferably kept short. Incentives can be offered, for example, a small amount of money with a postal questionnaire. One or more reminders can be sent with a copy of the questionnaire to people who do not reply.
Critical/reflective thinking: Imagine that you were asked to participate in a survey. Which of these strategies do you think would motivate or remind you to respond and why?
Quasi-experiments are a lot of work, yet don’t have the same scientific powerto show cause and effect, as do randomized controlled trials (RCTs).An RCT would provide better support for any hypothesis that X causes Y. [As a quick review of what quasi-experimental versus RCT studies are, see “Of Mice & Cheese” and/or “Out of Control (Groups).”]
So why do quasi-experimental studies at all? Why not always do RCTs when we are testing cause and effect? Here are 3 reasons:
#1 Sometimes ETHICALLY the researcher canNOT randomly assign subjects to a control and an experimental group. If the researcher wants to compare health outcomes of smokers with non-smokers, the researcher cannot assign some people to smoke and others not to smoke! Why? Because we already know that smoking has significant harmful effects. (Of course, in a dictatorship, by using the police a researcher could assign them to smoke or not smoke, but I don’t think we wanna go there.)
#2 Sometimes PHYSICALLY the researcher canNOT randomly assign subjects to control & experimental groups. If the researcher wants to compare health outcomes of
individuals from different countries, it is physically impossible to assign country of origin.
#3 Sometimes FINANCIALLY the researcher canNOT afford to assign subjects randomly to control & experimental groups. It costs $ & time to get a list of subjects and then assign them to control & experimental groups using random numbers table or drawing names from a hat.
Thus, researchers sometimes are left with little alternative, but to do a quasi-experiment as the next best thing to an RCT, then discuss its limitations in research reports.
Critical Thinking: You read a research study in which a researcher recruits the 1st 100 patients on a surgical ward January-March quarter as a control group. Then the researcher recruits the 2nd 100 patients on that same surgical ward April-June for the experimental group. With the experimental group, the staff uses a new, standardized pain script for better pain communications. Then the pain communication outcomes of each group are compared statistically.
Is this a quasi-experiment or a randomized controlled trial (RCT)?
What factors (variables) might be the same among control & experimental groups in this study?
What factors (variables) might be different between control & experimental groups that might affect study outcomes?
How could you design an ethical & possible RCT that would overcome the problems with this study?
Why might you choose to do the study the same way that this researcher did?
In the last “Quasi-wha??” blogpost, I described 1 type of experimental design: Quasi-experimental. To review… In quasi-experimental designs, the researcher manipulates some variable, but either 1) doesn’t randomly assign subjects to a control and experimental group OR 2) doesn’t have a control group at all.
For example, the researcher may introduce pet therapy on unit #1 and avoid pet therapy on unit #2 and then afterwards compare the anxiety levels of patients on the 2 units. That study has a control group (unit #2), but because patients weren’t (& probably couldn’t be) randomly assigned to the units, this would be a quasi-experimental study. The control group in this pet therapy case is what researchers call a “non-equivalent control group.” Non-equivalent means the groups are different in ways that might affect study results! [Note: For review of what constitutes a true experimental study see first part of “Quasi-wha??” blogpost.]
Herein lies a weak link in the cause-and-effect chain. Quasi- designs are NOT as strong as true experimental designs because something other than our treatment (in this case pet therapy) may have created any difference in outcomes (e.g., anxiety levels). Why? Here’s your answer.
In an experimental study, randomly assigning subjects to a
control and a separate experimental group means that all the little, variable weirdities of all subjects are equally distributed to each group. Each group is the same mix of different types of people. This means we can assume that both groups are the exact same type of
people in regard to things that may influence study outcomes, such as attitudes, values, preferences, beliefs, anxiety level, psychology, physiology and so on.
In contrast, in the quasi-experimental pet therapy example above, there is probably something that caused a certain type of person to be on unit #1 and a different type to be on unit #2. Maybe it was their diagnosis, their doctor, their type of surgery, or other. Thus, we cannot assume that people in unit #1 and unit #2 groups are the same before pet therapy, and so any differences between them after pet therapy might have already existed.
So why do quasi-experimental studies at all?? There are great reasons! Stay tuned for next blogpost.
If you haven’t done a scientific research study before or don’t have a PhD, then realize that your project will go much more smoothly if you consult with a PhD or someone with experience.
You bring the great clinical ideas, & the experienced researcher will bring research design expertise. The design is the overall research plan for getting and analyzing the data to answer your question or to find out how well your new ideas work. That person will know the technical things you need to plan into your study in order to make the study ‘sparkle’ and to get approval from human subjects review committees. The person doesn’t have to be an expert on your topic. You fill that role, or soon will!
If you have access to a librarian who is good at helping you look for current literature, s/he is one of your Best friends in getting a project done.
Searching for on-target literature from the millions of publications out there takes some special skills. Of course you can learn these on your own, but how much nicer to talk with a librarian about the key ideas in your project and allow them to use their special skills to help you. As an experience researcher, I can tell you that good librarians are worth their weight in gold! Librarians can help you find what others have learned about your topic already, and then you can build on that knowledge. [note: check out Finding the Needles in the Haystacks: Evidence Hunting Efficiently & Effectively for more]
Because it’s your first foray into research, you might want to stick with a descriptive study.
What does that mean? It means that you will collect data about what the current situation is. For example, you might measure the average days to return of bowels sounds on your unit, OR the number of minutes it takes to do some task, OR the interruptions of patient sleep during the night. This will help you to establish whether or not there really is a problem to be solved. Descriptive studies are much simpler to conduct and analyze than experimental studies in which you measure something, make an improvement, and then see if the improvement improved things. For example, you would measure sleep interruptions, institute a quiet time, and then measure sleep interruptions again to see if there were fewer. [check out “What it is.” – a primer on descriptive studies for more]
Pick a topic you are really jazzed about!
Every researcher from time to time can feel ‘bogged down’ or bored with what they are doing, & one of the best protections against that is making sure you think the topic is super-interesting in the first place. If you get a little bored or stuck later don’t be surprised; it just means you’re pretty normal. Those stuck times might even feel like “hitting the wall” in a long race, and once you get past it things get better. Remind yourself why you loved the topic in the first place. Talk to your PhD friend or a mentor for encouragement. Take a little break. Read something really interesting about your topic.
While not every step of the research study process will make you want to jump up, sing, and dance, the process as a whole is really rewarding and great fun. You will be empowered by new learning—not just about your topic, but about how to do research!