This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Glazerman, S. Articles by Myers, D. Search for Related Content Social Bookmarking                         What's this?

Nonexperimental Versus Experimental Estimates of Earnings Impacts

Mathematica Policy Research

Dan M. Levy

Mathematica Policy Research, U.S. Centers for Disease Control and Prevention

David Myers

Mathematica Policy Research

To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.

Key Words: nonexperimental • quasi-experimental • design replication

The ANNALS of the American Academy of Political and Social Science, Vol. 589, No. 1, 63-93 (2003)
DOI: 10.1177/0002716203254879


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				E. A. Stuart and D. B. Rubin Matching With Multiple Control Groups With Adjustment for Group Differences Journal of Educational and Behavioral Statistics, September 1, 2008; 33(3): 279 - 306. [Abstract] [Full Text] [PDF]

				D. A. Freedman and R. A. Berk Weighting Regressions by Propensity Scores Eval Rev, August 1, 2008; 32(4): 392 - 409. [Abstract] [PDF]

				K. A. Frank, G. Sykes, D. Anagnostopoulos, M. Cannata, L. Chard, A. Krause, and R. McCrory Does NBPTS Certification Affect the Number of Colleagues a Teacher Helps With Instructional Matters? Educational Evaluation and Policy Analysis, March 1, 2008; 30(1): 3 - 30. [Abstract] [Full Text] [PDF]

				D. E. Ho, K. Imai, G. King, and E. A. Stuart Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference Political Analysis, July 1, 2007; 15(3): 199 - 236. [Abstract] [Full Text] [PDF]

				K. Arceneaux, A. S. Gerber, and D. P. Green Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment Political Analysis, January 1, 2006; 14(1): 37 - 62. [Abstract] [Full Text] [PDF]

				J. K. Luellen, W. R. Shadish, and M. H. Clark Propensity Scores: An Introduction and Experimental Test Eval Rev, December 1, 2005; 29(6): 530 - 558. [Abstract] [PDF]

				L. W. Sherman and H. Strang Experimental Ethnography: The Marriage of Qualitative and Quantitative Research The ANNALS of the American Academy of Political and Social Science, September 1, 2004; 595(1): 204 - 222. [Abstract] [PDF]

				L. W. Sherman and H. Strang Verdicts or Inventions?: Interpreting Results from Randomized Controlled Experiments in Criminology American Behavioral Scientist, January 1, 2004; 47(5): 575 - 607. [Abstract] [PDF]