Why software engineering researchers should engineer software
The research field of software engineering is very young. Compared to other research fields its in its infancy even. What defines this field is a search for better software and a search for better ways to construct and maintain software. As a research field it is not just about finding solutions to engineering problems instances, but rather about theory forming (to enable effective problem solving for a wider range of problem instances) and validation of these theories, and then constructing tools based on these (possibly validated) theories to help spreading these solutions or to make them feasible at all. Possibly these things do not happen in the previous order, which can be fine.
Unfortunately, the validity of research output is extremely hard to assess in the field of software engineering. This is due to many things, our youth not being the least. Because the field has yet to establish well-defined research methods, and even well-defined and agreed upon research questions, the validity of each result is “loose on both ends” so to speak. It’s not just the validity of the result which must be discussed in every paper, but rather the framework against which it is compared to check its validity. Frameworks to compare to would be for example:
- standardized research methods for typical research questions (like comparison trial against a placebo in medicine)
- benchmarks (well-defined problems to test new solutions against)
- base rates (statistical data on factors which impact software in general)
In my experience, most papers usually come both with a new frame of reference (at least partially) as well as the new results of the current research question. It is just the way it is, and time will tell when we grow out of this infancy stage as a research field. Now do not get me wrong, sometimes new frames of reference are the only way to break new ground. But the incremental work we do should not be so shifty; otherwise we would never even recognize when real ground is broken among the ample noise of our widespread confusion.
Next to this lack of established research question types and respective research methods, the field is enormously complex as well as vaguely defined at the boundaries. This means that many factors come into play when we talk about “better software” and “better software engineering”, including mathematics, physics, psychology, sociology; all of these factors in motion and under evolution at the same time. Well, it’s indeed an interesting field and that is why we love to study it.
The novelty and complexity of software engineering together allow the possibly naive researcher to:
- pick interesting-sounding but otherwise arbitrary research questions from the pile;
- select “common sense” research methods to answer these questions;
- produce interesting written results based on the previous.
The main problem with this strategy is that experienced software engineers will cry “BS” when they read the paper. First, if we as researchers are not experienced engineers ourselves, we can actually not recognize without advise what a good research question is. Second, our method being based on “common sense” is in dire need of sceptical scrutiny, because we do not have the ground truth to compare our results to and we have not ironed out the teething troubles of yet another new research method.
Two things researchers in software engineering should do is publish their research prototypes as open-source software and immerse themselves in the activity of software engineering. The reason for the first is that software lends itself perfectly for sharing, especially if its government funded software. There is no excuse not to do this. The reason for the second is that software engineering is so wickedly complex and rapidly evolving, that without doing it yourself it is easy to misunderstand what the problems are or to recognize good solutions.