Designing Social Inquiry
Designing Social Inquiry: Scientific Inference in Qualitative Research is an influential 1994 book written by Gary King, Robert Keohane, and Sidney Verba that lays out guidelines for conducting qualitative research. The central thesis of the book is that qualitative and quantitative research share the same "logic of inference". The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research.
The text is often referred to as KKV within social science disciplines. The book has been the subject of intense debate among social scientists. The 2004 book Rethinking Social Inquiry, edited by Henry E. Brady and David Collier, is an influential summary of responses to KKV.
Contents
The goal of the book is guide researchers in producing valid causal inferences in social science research. The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research. The authors argue “whether we study many phenomena or few… the study will be improved if we collect data on as many observable implications of our theory as possible.” The authors note that case studies do not necessarily have to be N=1 or few N: a case study can include many observations within a case. KKV criticize Harry H. Eckstein's notion of "crucial case studies", warning that a single observation makes it harder to estimate multiple causal effects, more likely that there is measurement error, and risks that an event in a single case was caused by random error.According to the authors, a strong research design requires both qualitative and quantitative research, a research question that poses an important and real question that will contribute to the base of knowledge about this particular subject, and a comprehensive literature review from which hypotheses are then drawn. Data that are collected should be operationalized so that other researchers could replicate the study and achieve similar results. For the same reason, the reasoning process behind the analysis needs to be explicit. While gathering data the researcher should consider the observable implications of the theory in an effort to explain as much of the data as possible. This is in addition to examining the causal mechanisms that connect one variable to another.
While qualitative methods cannot produce precise measurements of uncertainty about the conclusions, qualitative scholars should give indications about the uncertainty of their inferences. KKV argue that "the single most serious problem with qualitative research in political science is the pervasive failure to provide reasonable estimates of the uncertainty of the investigator’s inferences."
According to KKV, the rules for good causal theories are that they need to be falsifiable, have internal consistency, have variation, have "concrete" concepts, and have "leverage".
KKV sees process-tracing and qualitative research as being "unable to yield strong causal inference" due to the fact that qualitative scholars would struggle with determining which of many intervening variables truly links the independent variable with a dependent variable. The primary problem is that qualitative research lacks a sufficient number of observations to properly estimate the effects of an independent variable. They write that the number of observations could be increased through various means, but that would simultaneously lead to another problem: that the number of variables would increase and thus reduce degrees of freedom.
In terms of case selection, KKV warn against "selecting on the dependent variable". For example, researchers cannot make valid causal inferences about wars outbreak by only looking at instances where war did happen. There is methodological problem in selecting on the explanatory variable, however. They do warn about multicollinearity. They argue that random selection of cases is a valid case selection strategy in large-N research, but warn against it in small-N research.
KKV reject the notion of "quasi-experiments", arguing that either all the key causal variables can be controlled or not.
Reception
In a 2010 review by James Mahoney of prominent methods books, he writes that critics of KKV has characterized the book's claims as "often simplistic, misleading and inappropriate as a guide for designing social inquiry." Quantitative scholars such as Henry E. Brady, Larry M. Bartels and David A. Freedman have argued that KKV overstate the strengths of quantitative research vis-a-vis qualitative research. In his review, Mahoney writes that the field of social science methodology has "benefited from KKV even as it has also moved beyond it."Mahoney writes that KKV ignore set theory and logic in terms of evaluating causal inference. Whereas regression-oriented analyses seek to estimate average effects of certain outcomes, qualitative research seeks to explain why cases have certain outcomes. Thus, causal inference is not strengthened by expanding the size of N, but rather by carefully choosing cases, whose testing can strengthen or weaken a theory.
Mahoney also writes that KKV give insufficient attention to concept formation, which is an essential aspect of theory construction and measurement, and one of the important ways that qualitative research can play a key role.
Ronald Rogowski criticizes how KKV treat qualitative social science research. Rogowski argues that there is too much focus on hypothesis-testing and too much caution against using single observations. Rogowski argues that KKV promotes a form of qualitative social science that is overly focused on hypothesis-testing, and that this limits scholars' questions, cases and ambitions. John J. Mearsheimer and Stephen M. Walt argue that International Relations scholarship has shifted away from crafting and refining IR theory to "simplistic hypothesis-testing", in part due to the influence of KKV in political science graduate programs.
Alexander George and Andrew Bennett say there is "much to agree with" in KKV, but they argue that the book has several flaws in its guidance on qualitative research:
- Causal mechanisms: KKV suggest that "causal mechanisms" are less important that "causal effects" in causal explanations – George and Bennett argue that they are equally important
- Hypothesis-testing: KKV overly emphasize the role of hypothesis-testing in theory development – George and Bennett argue that the formation of new hypotheses and the choosing of new questions are also important aspects of theory development
- Causal complexity: KKV fail to consider problems of causal complexity, such as equifinality, multiple interaction effects, feedback loops, path dependency, tipping points, selection effects, expectations effects and sequential interactions – George and Bennet argue that case studies, process-tracing and typological theories can clarify causality in situations of causal complexity
- Increasing N: KKV argue that scholars should always seek to increase the number of cases – George and Bennett argue that KKV fail to consider the costs to increasing the number of cases. George and Bennett note that much value can be derived from single-case studies.
- Process-tracing: KKV characterize process-tracing as a way to increase the number of observable implications – George and Bennett argue that the logic of process-tracing is entirely different. The logic behind using process-tracing is to focus on the sequences and timings within a case, not to correlate data across cases. Thus, if one piece of evidence in the sequence is inconsistent with the theoretical expectations, then the theory has been shown to be flawed.
- "Degrees of freedom" problem: KKV argue that a single case cannot evaluate competing explanations due problems that arise from degrees of freedom – George and Bennett argue that it is flawed to apply this statistical logic to qualitative research. George and Bennett say that while quantitative scholars try to aggregate variables to reduce the number of variables and thus increase the degrees of freedom, qualitative scholars intentionally want their variables to have many different attributes and complexity.