“Measure twice. Cut once!” goes the old carpenter adage. Why? Because measuring accurately means you’ll get the outcomes you want!
Same in research. A consistent and accurate measurement will get you the outcomes you want to know. Whether an instrument measures something consistently is called reliability. Whether it measures accurately is called validity. So, before you use a tool, check for its reported reliability and validity.
A good resource for understanding the concepts of reliability (consistency) and validity (accuracy) of research tools is at https://opentextbc.ca/researchmethods/chapter/reliability-and-validity-of-measurement/ Below are quoted Key Takeaways:
- Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.
- There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (interrater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.
- Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.
- The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.
Reliability & validity are terms that refer to the consistency and accuracy of a quantitative measurement questionnaire, technical device, ruler, or any other measuring device. It means that the outcome measure can be trusted and is relatively error free.
- Reliability – This means that the instrument measures CONSISTENTLY
- Validity – This means that the instrument measures ACCURATELY. In other words it measures what it is supposed to measure and not something else.
For example: If your bathroom scale measures weight, then it is a valid measure of weight (e.g. it doesn’t measure BP or stress). You might say it had high validity. If your bathroom scale measures your weight as the same thing when you step on and off of it several times then it is measuring weight reliably or consistently; and you might say it has high reliability.
Self-report by participants is one of the most common ways that researchers collect data, yet it is fraught with problems. Some worries for researchers are: “Will participants be honest or will they say what they think I want to hear?” “Will they understand the questions correctly?” “Will those who respond (as opposed to those who don’t respond) have unique ways of thinking so that my respondents do not represent everyone well?” and a BIG worry “Will they even fill out and return the questionnaire?”
One way to solve at least the latter 2 problems is to increase the response rate, and Edwards et al (2009 July 8) reviewed randomized trials to learn how to do just that!!
If you want to improve your questionnaire response rates, check it out! Here is Edwards et al.’s plain language summary as published in Cochrane Database of Systematic Reviews, where you can read the entire report.
Methods to increase response to postal and electronic questionnaires
Postal and electronic questionnaires are a relatively inexpensive way to collect information from people for research purposes. If people do not reply (so called ‘non-responders’), the research results will tend to be less accurate. This systematic review found several ways to increase response. People can be contacted before they are sent a postal questionnaire. Postal questionnaires can be sent by first class post or recorded delivery, and a stamped-return envelope can be provided. Questionnaires, letters and e-mails can be made more personal, and preferably kept short. Incentives can be offered, for example, a small amount of money with a postal questionnaire. One or more reminders can be sent with a copy of the questionnaire to people who do not reply.
Critical/reflective thinking: Imagine that you were asked to participate in a survey. Which of these strategies do you think would motivate or remind you to respond and why?
For more info read the full report: Methods to increase response to postal and electronic questionnaires
“What’s important is not where an organization begins its patient safety journey, but instead the degree to which it exhibits a relentless commitment to improvement.” – TJC, 2016, p.68
The path to zero harm, according to TJC, begins with high reliability. Reliability in research = consistency. TJC says for zero harm we as providers must be consistent in these ways:
- Never be satisfied with your safety record. Always be alert for danger
- Be alert for early signs of potential danger. Don’t oversimplify your observations
- Note small changes in the organization as having longer range or unintended effects
- Commit to resilience so that when errors do happen, you bounce back quickly
- When confronted by a threat, put its resolution in the hands of those with the most expertise in that area
Using evidence in practice can be part of our “relentless commitment to improvement,” especially when coupled with above 5 actions and can support zero harm to patients. That evidence can be from research, from process improvement, from evaluation of clinical innovations, or from experts.
For more read TJC’s High Reliability: The Path to Zero Harm online at http://www.jointcommission.org/assets/1/18/HC_Exec_article.pdf