Why Better Methods Cannot Save Bad Questions

Psychology’s recurring reform movements share a common hope: that methodological improvement will resolve epistemic weakness. Larger samples, cleaner designs, preregistration, better statistics, and open data are repeatedly invoked as correctives. Each reform addresses real problems. None can rescue a poorly posed question. When inquiry begins from a confused, trivial, or misaligned question, methodological sophistication amplifies the problem rather than solving it.

A research question does more than initiate a study. It defines what will count as relevant evidence, which variables will be treated as meaningful, and what kinds of answers are even possible. Method operates downstream of these decisions. If the question is conceptually thin, no amount of rigor can thicken it. At best, better methods produce precise answers to questions that never deserved to be asked in the first place.

This problem is not new. Paul Meehl’s critique of psychological research emphasized that weak theories generate weak questions, which in turn generate ambiguous findings regardless of statistical care. Meehl’s concern was not primarily technical. It was conceptual. When theories lack constraints, questions become permissive, capable of being “answered” affirmatively under many conditions. Improved methods then give such answers a misleading air of authority.

Psychology’s institutional structure encourages this dynamic. Funding agencies, journals, and graduate training reward feasibility, not depth. Questions that can be operationalized quickly, tested efficiently, and reported cleanly are favored. Questions that require prolonged conceptual development or challenge foundational assumptions are riskier. Over time, the field becomes populated with studies that are methodologically polished and conceptually modest.

By the time I entered the field in the mid-1980s, this pattern was already visible. Methodological competence was emphasized early and often. Question formation was treated as a preliminary step rather than as an intellectual achievement in its own right. Students learned how to test hypotheses before learning how to generate ones worth testing.

The result is a literature dense with answers and sparse in insight. Many studies ask whether a variable is related to another variable under specified conditions. Far fewer ask why the relationship should exist, what it presupposes about human functioning, or how it fits within a broader theoretical landscape. The accumulation of such findings creates the illusion of progress while leaving core questions untouched.

Bad questions often share recognizable features. They are underspecified, relying on intuitive constructs that have not been conceptually clarified. They are trivial, confirming what is already obvious under laboratory conditions. Or they are misaligned, importing questions from other disciplines without adapting them to psychology’s subject matter. In each case, better methods increase confidence without increasing understanding.

The history of social priming research illustrates this point starkly. Many studies were methodologically competent by the standards of their time. Yet the questions they posed assumed implausibly direct links between subtle cues and complex behavior without adequate theoretical grounding. When replication failed, methodological explanations dominated the response. What was less frequently acknowledged was that the original questions overestimated the coherence and strength of the proposed mechanisms.

Cronbach’s distinction between experimental and correlational psychology is relevant here as well. He warned that technical refinement within one mode of inquiry could not substitute for theoretical integration across modes. Asking better questions requires knowing which mode is appropriate, and why. Methods answer questions within a frame. They do not select the frame.

The emphasis on better methods also encourages a form of displacement. Conceptual difficulty is reinterpreted as technical challenge. Instead of asking whether a construct is coherent, researchers ask how to measure it more precisely. Instead of asking whether a question is meaningful, they ask how to power the study adequately. The difficulty shifts, but the underlying problem remains.

This displacement is reinforced by the moral language surrounding rigor. Methodological improvement is treated as ethical progress, while conceptual critique is treated as obstruction. It is easier to call for preregistration than to argue that a dominant research program rests on shallow questions. The field learns to fix procedures rather than to interrogate its aims.

None of this diminishes the importance of method. Psychology requires technical competence to avoid self-deception. But competence is not insight. Methods discipline inquiry; they do not originate it. The origin lies in judgment about what is worth asking, which phenomena deserve sustained attention, and which distinctions matter psychologically.

A discipline that neglects question formation becomes reactive. It follows available tools, fashionable constructs, and fundable topics. Over time, its agenda reflects opportunity more than understanding. Better methods then serve to optimize productivity rather than deepen knowledge.

Recovering the centrality of questions requires slowing down a process designed for speed. It requires training students to treat question formation as a scholarly act, not as a prelude. It requires rewarding work that clarifies concepts, articulates mechanisms, and identifies genuine uncertainty. These activities are harder to quantify than methodological compliance, but they are what determine whether a field advances.

Psychology does not suffer from a lack of answers. It suffers from an excess of answers to questions that do not reorganize understanding. Better methods will continue to matter. But without better questions, they will continue to produce confidence without clarity.

The discipline’s future credibility depends less on how rigorously it answers its questions than on whether those questions are worthy of the rigor applied to them.

Letter to the Reader

If you have ever felt that a study was impeccably designed yet oddly unilluminating, trust that reaction. When I look back over decades in the field, what stands out is not how often methods failed, but how often questions were too small for the phenomena they targeted.

Learning psychology well means learning to ask questions that risk being wrong in meaningful ways. Methods protect us from error. Questions determine whether the answers matter.