THE DESIGN used in this study1–quasi-experimental–is
unfamiliar to clinical readers. There are few easily available worksheets
or prescriptive methods for reviewing such an article. We used 2 excellent
resources to generate questions used in the evaluation of this article,2, 3 modified by 2 of us (M.H. and W.B.),
and reordered here to match the sequence of the study. The graphic (Figure 1) helps to arrange the methodological
issues we raise throughout the article. In the Web-based version we provide
links, depicted in the graphic, to provide further detail about the various
methodological considerations and threats to validity.
Figure appears in full text version.
|
|
|
|
Study flow diagram. Each study phase engenders specific biases,
and provides the basis for the discussion.
|
|
|
PRIOR TO THE STUDY
Was the Problem Clearly Stated?
Yes. There have been few long-term evaluation studies of educational
interventions to reduce the high rates of pregnancy and sexually transmitted
diseases (STDs) (including human immunodeficiency virus) in adolescents in
the United States. These public . . . [Full Text of this Article]
What
Design
Was Used in This Study? Is This Design Experimental
or Quasi-Experimental?
Are the Hypotheses Chosen by the Authors Intuitive and Reasonable?
(Minimize
Threats due to Inadequate Conceptual Model)
POPULATION AND GENERALIZABILITY
Did the Authors Accurately Define the Target Population? Were
Attempts Made to Select a Representative Sample?
Was the Intervention Replicated in a Similar Setting?
(Minimize
Threats due to Generalizability)
RECRUITMENT AND BASELINE ASSESSSMENT
Were Detailed Demographic Characteristics of the Participants Obtained?
Was There an Adequate Determination of Base Rates Used in Any
Independent or Dependent Variable?
Did the Authors Delineate All Factors That Might Be Related to Outcome? Did
They Choose Reasonable, Appropriate, and Independent Variables?
Were Pretest Measures Used to Detect a Pattern or Trend Prior to the
Intervention? (Minimize the Effect of
Maturation Bias)
ASSIGNMENT
Was Random Assignment
to Intervention Groups Used?
Did the
Sample Size
Used Have Adequate
Power
to Detect a Difference
Between Intervention and Comparison Groups?
Were Any Differences in Group Characteristics Found on Pretesting Adjusted
for in Statistical Analysis? (Minimize Threats due to
Selection Bias)
INTERVENTION
Were Multiple Pilot Tests or Pretest Observations Obtained? (Minimize
Threats due to
Instrumentation or Interaction With Selection That Differentiate the Groups)
Was the Intervention Implemented Consistently and Reliably?
Was Contact Between Intervention and Control Groups Minimized? (Minimize
Threats due to
Treatment Contamination, Atypical Responses, and Reporting Bias)
Was a
Process Evaluation
Used to Monitor the Administration of the
Intervention? Was the Intervention Process Evaluated Systematically?
Were Attempts Made to Minimize Data Recording Errors
(Such as
Double Data Entry)?
OUTCOMES
Did the Authors Specify the Expected Outcomes?
Were the Dependent Measures Used Valid and Reliable?
Were Multiple Indicators Used to Assess Each Dependent and Independent
Variable? (Minimize Threats due to
Unreliable Operational Definitions, Poor Generalization of Construct to Other Constructs)
ANALYSIS
Were Appropriate Statistical Tests Used?
Did the Authors Use Baseline Assessment to Effectively Evaluate If
Their Intervention Improves Behavior and/or Knowledge?
Do the Authors Satisfactorily Address the Alternative Hypotheses That
Might Explain the Observed Effect?
OVERALL
Would You Recommend That the Editor Accept This Article for Publication?
What Additional Information Would You Request From the Authors for
Revisions?
CONCLUSION
FURTHER READING
