Threats to External Validity: In a nut shell this has to do with our ability to say if the researchers' findings would be useful with a similar group of people. This is called generalizability. Important: If the participants were adolescent girls then we would be thinking: "Do I think this applies to other adolescent girls?" Not everybody can be studied so it isn't fair to criticize the researchers because they didn't include adolescent boys!

I have bolded the key ones!

Threat What does this mean? What kind of study is most likely to effected?
Small sample size Bigger is not always better, but you want enough people in the study so you feel comfortable believing they represent other people like them. Convenience samples where there are a smallish (< 30) number of participants.

Reactive effects of testing

 

The study results may not be generalizable to people who didn't undergo a pretest. This would be related to the internal validity threat of testing. That alone may have raised awareness.

Any study with a pre-test. For example if you were in a study related to nutrition then a pre-test might be what you had eaten in the last couple of days. The "pre-test" might have reminded you to eat healthier before you even started the study!

Interaction effects of selection and the independent variable

 

The effects of the internal threats of selection bias and receiving the treatment may compound each other making it difficult to discern how much is treatment and how much is the participants' characteristics.

 

 

 

Refusal rates: This is the percentage of people who agreed to be in the study as opposed to those asked to be in the study. The researcher needs to report the refusal rate. Anything less than 80% makes the generalizability suspect.

Studies that require a major time commitment or inconvenience to the participants. Those who chose to participate may for example, be hardier, more gullible or have more time on their hands than the general population! Also if choosing participants was difficult and there was a high refusal rate than the participants may be different.

Studies requiring a big time commitment by the participants even if it is only a one shot deal. It helps if the researcher has collected some basic demographics on those who refused and then comparisons can be run between the groups to see if they are different or similar. Also if researchers get a low response rate, they need to asked themselves "why has this occurred?" Is the questionnaire too long? Is it a subject of little interest to the population being studied?

Interaction effects of setting and the independent variable

Where the study took place may effect how well the intervention worked. For example some hospitals or clinics or schools might be more willing to allow nurse researchers to conduct studies and the types of patients going to those hospitals may be different than hospitals less willing to participate in studies. Think UCSF and Stanford!

The trade-off is if the study is done in the same hospital and the same time there is a major risk of contamination (an internal validity threat)!

This would be another plug for replication studies. The same study design could be sued with different populations.

Studies related to the same phenomenon which used similar types of hospitals.

In a study I conducted with pregnant adolescents and their intention to continue in school after the birth of their children, I used schools specifically for pregnant and parenting teens. I had sound reasons for doing this decision, but it meant study results could only be generalized to girls in similar schools.

Interaction of history and the independent variable

When the study took place is always a consideration and results can't be generalized across time. State of the art equipment changes as do the understandings about adverse physiologic conditions. I am such a long time nurse that I can remember when we treated all 9 pound babies as term babies! Oops! Also think of the political climate. Women who had abortions 35 years ago might have very different outcomes then women having abortions today.

Studies in which the intervention is likely to have changed or the targeted population may have changed. Studies related to domestic violence conducted 20 years ago often included only poor women in shelters. Now there is a bit more diversity in the targeted population.

Again, replication studies would be beneficial!

Reactive effects of experimental arrangements

 

Participants may react to being in a study and that alone can influence the study results. Factors such as novelty, the Hawthorne effect (people change their behavior because they are in a study) or the placebo effect are included.

Studying participants over a short period of time. Of course this has trade-offs. The longer the study goes on, the more likely people will return to their previous behaviors when the novelty has worn off. This could be a factor in the internal threat of statistical regression.

Also participants in an experimental study may be followed much more closely then people would be in general (think adherence to medication regimes)

 

 

 

 
Jeanette Koshar, RN, NP, PhD
Office: (707) 664-2649 | Office Hours: Wednesday 12-3, email and by appointment
Email:
jeanette.koshar@sonoma.edu