Threats to Internal Validity in Quantitative Studies. In a nutshell Internal Validity has to do with how the study was set up and conducted including how the participants were selected and "managed: once they were in the study!

I have bolded the key ones!

Threat What does this mean? What kind of study is most likely to effected?
Temporal Ambiguity

Is the independent variable really coming before the dependent variable?

Do sweat pants make you fat?

Comparative and correlational designs (especially if they are not prospective)

History

 

An event that occurs during a study that can effect the responses of the participants. It could be something the participants do like start exercising all on their own or it could be an event such as an article in the newspaper or national publicity.

Longitudinal. A study with repeated measures.

Maturation

 

The participants get older, wiser, more depressed, more hungry, more tired. They change all on their own! Longitudinal studies where the participants are more likely to change such as adolescents, infants or people who are quite ill.

Testing

Sometimes the effect being measured changes because of the number of times the participant is tested. The test itself may add to the person's knowledge or change their attitudes, hence the test becomes part of the intervention! Studies with repeated measures or a pre-test-post test design.

Instrumentation

This means changes in the instrument during the study. It can also mean the data collectors get better or worse at what they are doing. To me this always seems like a reliability issue, but it is always listed under internal validity.

Physiologic instruments or those in which researchers are collecting the data in person.

Statistical regression In general, participants who score really high or really low on a test or a questionnaire will have a more moderate score the next time they are tested. This is called regressing towards the mean. Pretest-posttest designs. A study with a small sample size will have more of an effect since with less participants, one person's scores has a greater effect on the average score.
Selection Pre-existing differences between the participants selected for a study or those who volunteer for a study and those not in the study or differences between or among the study groups. This is usually called selection bias. Quasi-experimental studies and convenience samples. Doing statistical comparisons between groups helps address this threat. For example are both groups similar ages, ethnicity, SES, severity of their illnesses.
Mortality/Attrition No, this doesn't mean the sicker people die! Participants drop out of a study or are lost to follow-up. Longitudinal studies, This can especially be a problem if the participants have to make a big commitment to be in the study (it takes a fair amount of their time or effort). It is also a bigger problem if the participants tend to be more transient. It helps to have several contact phone numbers or addresses for each participant. It is important to analyze the existing data to determine if there are differences between those who dropped out and those who continued. If more than 10% of participants are lost to follow-up this seriously affects generalizability.
Interaction effect One or more of these threats can combine and have a compounding effect. The three most likely interactions are history, maturation and instrumentation.
Diffusion of treatment "Contamination" of the no-treatment group by the treatment group in an experimental or quasi-experimental study. Somehow the no-treatment group finds out what the treatment is and start doing it! It may also mean that the researchers change one of the intervention groups or control group to contain treatments not originally in their intervention. Studies with an intervention when the intervention groups and control groups are being studied at the same time in the same location (a clinic, small town). This means there is basically not a real control group any moore. I would consider this a fatal flaw! The trade off is the effects of history and selection bias.
Compensatory rivalry of the no-treatment group Participants know they are in the no-treatment group and so attempt to improve on their own. Good for them, but bad for the study! Intervention studies where the participants are aware they are in the no treatment group. It can also occur if the health care providers know the participants are in the no-treatment group so they give better care. Blinded and double blinded studies help.
Resentful-demoralization of the no-treatment group The opposite of above. The control group gives up because they aren't getting the treatment. The reverse of the above.

Note: One cited web site lists these as internal validity threats and they are usually included under threats to external validity.

 

Experimenter bias, Placebo Effect, Hawthorne Effect.

 

 

 
Jeanette Koshar, RN, NP, PhD
Office: (707) 664-2649 | Office Hours: Wednesday 12-3, email and by appointment
Email:
jeanette.koshar@sonoma.edu