Provocatorio, divertente e... molto attinente alla realtà. https://twitchard.github.io/posts/2019-10-13-software-development-and-the-fa... # The Limits of Science Why is the project to construct a science-based discipline of software development hopeless? For one, I don’t think researchers are really aiming to do this. They are quite content to continue telling insightful stories about their relatively narrow specialties – bits of our discipline, here and there. There is no grand plan to reform us. But even if there were, it would be bound to fail for methodological reasons. This is worth some discussion, so I’ll describe four of what I mean by “methodological reasons”. [...] 1. Controlled experiments are typically nothing like professional programming environments [...] 2. The unpredictable dynamics of human decision-making obscure the effects of software practices in field data. [...] 3. The outcomes that can be measured aren’t always the outcomes that matter. [...] 4. Software practices and the conditions which modify them are varied, which limits the generality and authority of any tested hypothesis [...] # Science is not the only source of knowledge Is this cause for despair? If science-based software development is off the table, what remains? Is it really true as Hillel suggests, that in the absence of science “we just don’t know” anything, and we are doomed to an era of “charisma-driven development” where the loudest opinion wins, and where superstition, ideology, and dogmatism reign supreme? Of course not. Scientific knowledge is not the only kind of knowledge, and scientific arguments are not the only type of arguments. Disciplines like history and philosophy, for instance, seem to do rather well, despite seldom subjecting their hypotheses to statistical tests. [...] Software development, like history and philosophy, can be understood without measurement. [...] And here is a secret: in the natural sciences themselves, storytelling and bare conjecture are far more important modes of persuasion than data-based empirical argument, anyway. [...] In fact, progress in science can sometimes depend on scientists to proceed “counterinductively”, and, driven by intuition, ideology, insanity, or other sentiments, deliberately accept a theory that is less consistent with the data. [...] # Science gone wrong Science – or at least a mysticized version of it – can be a threat to this sort of inquiry. Lazy thinkers and ideologues don’t use science merely as a tool for critical thinking and reasoned argument, but as a substitute. Science appears to offer easy answers. [...] You can probably get away with just reading the abstracts, even. No need to trouble yourself about the line of argument that led the researchers to their conclusions. Heaven forbid you read the “threats to validity” section – nobody takes that seriously. They’ve got p-values, confidence intervals and such. Those are essentially SLAs on the objective truth. [...] No need to demonstrate the shortcomings and inconsistencies of their ideas. We can simply dismiss them as “anti-science” and compare them to anti-vaxxers. Why appeal to rational sensibilities when you can exploit tribal instincts instead? This sentiment exists. I witnessed it play out among industry leaders in my Twitter feed, the day after I started drafting this post. It’s amplified by the fact that nuanced scientific analysis doesn’t make good clickbait, and by the fact that everything except the abstract of a research paper tends to live behind a paywall. # TL;DR Software developers are domain experts. We know what we’re doing. We have rich internal narratives, and nuanced mental models of what it is we’re about, and we can learn to get better through the simple but difficult act of telling our stories, articulating our ideas and listening to each other, like philosophers or historians do. Empirical researchers have a different perspective and can say interesting things about our field, but it is important to consider these arguments on their (necessarily limited) merits, not idealizing them as an absolute authority. Down that road lies pettiness and lazy thinking.